Nemo v0.6.0

New Features and Improvements

• Major rework of the parser, allowing for more expressive error messages
• Improved Rule API
• Added REGEX function
• Added Unicode support for String functions
• Support Trace as DAG in GraphML format
• Boolean functions can now be used on the outer level in a rule body
• More accurate reporting of memory consumption
• Improve memory consumption for storing string data
• Provide timing info in the python api

Notable bux fixes

• Fix off-by-one error in SUBSTR function (#481)

Full Changelog: v0.5.1...v0.6.0

Nemo v0.5.1

New Features and Improvements

• More builtins, including: SUM, PROD, MIN, MAX, LUKA, BITAND, BITOR, BIXOR
• Some existing functions now allow arbitrary many arguments like CONCAT
• Alternative ternary version of SUBSTR added
• Automatically upcast numeric values if needed (e.g. adding an integer and a double automatically results in a double without the need to cast the integer)

Notable bug fixes

• Fixed a bug where the occurrence of multiple datatypes in a single relation could lead to wrong results (#477, #483)
• Fixed a bug that prevented an optimization in rules which restrict variables to constants (#487)
• Fixed a bug that prevented gzipped data from working when imported from an HTTP source (#486)

Full Changelog: v0.5.0...v0.5.1

Nemo v0.5.0

Version 0.5.0 of Nemo sees the fast in-memory rule reasoner with a redesigned data model that natively supports storing multiple data types. The comman-line client nmo can be used to access all functionality (use nmo --help for a brief documentation). The online documentation of Nemo covers many of the current features.

New Features and Improvements

• All supported RDF formats are now also available as output formats
• New built-in operations isInteger, isFloat, isDouble, isIri, isNumeric, isNull, isString, ABS, SQRT, NOT, fullStr, STR, SIN, COS, TAN, STRLEN, UCASE, LCASE, ROUND, CEIL, FLOOR, DATATYPE, LANG, INT, DOUBLE, FLOAT, LOG, POW, COMPARE, CONCAT, CONTAINS, SUBSTR, AND, OR, STRSTARTS, STRENDS, STRBEFORE, STRAFTER, and REM
• The predicates to trace can now be read from a file (--trace-input-file)
• Output can be directed to the standard output by specifying an empty resource, e.g., @export predicate :- csv{resource=""}

Notable bug fixes and internal improvements

• New data model handles data types natively with less memory overhead

Breaking changes

• @source directives are no longer supported, use @import instead
• @declare directives are no longer supported; appropriate types should be inferred automatically (or specified on @import)
• The --output-dir CLI option has been renamed to --export-dir
• The --input-dir CLI option has been renamed to --import-dir
• The --save-results CLI option has been replaced by the --export option, which takes the values none (do not export anything), keep (export predicates annotated with @output; this is the default behaviour), idb (export all IDB predicates), edb (export all EDB predicates), and all (export everything)
• The --write-all-idb-predicates CLI option has been removed in favour of --export idb
• The --detailed-memory and --detailed-timing options have been replaced by the --report option, which takes the values none (do not report anything), auto (print a short report if no other results are printed, otherwise do not print anything; this is the default behaviour), short (print a short report), time (print a short report and a detailed timing report), mem (print a short report and a detailed memory usage report), and all (print all reports)

Current Functionality

• Execution (materialization) of Datalog extended with stratified negation and existential rules (tuple-generating dependencies)
• Loading input data from local or remote CSV/TSV and RDF (NT, Turtle, RDF/XML, NQ, TriG) files
• Writing results to CSV files
• Compatibility with RDF and SPARQL syntax for IRIs and literals
• Datatypes integer (whole numbers), float64 (64bit floating point numbers), string (Unicode strings), and any (union type that can represent any element; default for most contexts)
• Support for built-in predicates <=, <, >=, >, !=, and = and built-in functions +, -, *, and / for types integer and float64
• Support for built-in functions isInteger, isFloat, isDouble, isIri, isNumeric, isNull, isString, ABS, SQRT, NOT, fullStr, STR, SIN, COS, TAN, STRLEN, UCASE, LCASE, ROUND, CEIL, FLOOR, DATATYPE, LANG, INT, DOUBLE, FLOAT, LOG, POW, COMPARE, CONCAT, CONTAINS, SUBSTR, AND, OR, STRSTARTS, STRENDS, STRBEFORE, STRAFTER, and REM
• Aggregation: #count, #min, #max, and #sum

Nemo v0.5.0 is built for mid-sized computing tasks that can still be processed on a normal laptop in seconds or minutes (typically hundreds of thousands to hundreds of millions of facts). In such cases, Nemo is already quite fast – at least fast enough to outperform existing free rule engines on the tests we conducted so far. Example tasks and benchmark results can be found in our sister repository Nemo examples and benchmarks.

Note that the combination of existential quantifiers and stratified negation do not have a standard semantics, and may lead to unexpected conclusions which can be avoided by careful modeling (see Ellmauthaler, Krötzsch, Mennicke; AAAI 2022).

Known Limitations

The following features are not included in v0.5.0 yet and will be added in upcoming releases:

• Support for unary - in arithmetic expressions (current workaround: use 0-?X instead of -?X)

Moreover, the documentation, though improved, is still very limited. This will be expanded gradually.

Full Changelog: v0.4.0...v0.5.0

Nemo v0.4.0

Version 0.4.0 of Nemo extends the fast in-memory rule reasoner with support for aggregation and tracing. The command-line client nmo can be used to access this functionality (use nmo --help for a brief documentation). The online documentation of Nemo covers many of the current features.

New Features and Improvements

• Aggregates in rule heads
• tracing of atoms shows why an atom was derived: e.g., the CLI option --trace 'p(42)' shows a derivation tree for atom p(42) after reasoning
• arithmetic terms can now be used in rule bodies
• inequality operator != for arithmetic comparisons
• wildcard terms: _ in an atom always matches, but does not introduce a variable binding, e.g., q(?x, ?y) :- p(?x, _, _, ?y) is equivalent to q(?x, ?y) :- p(?x, ?z, ?zz, ?y)
• loading of RDF N-Quads and TriG files
• @import syntax for more control over input formats
• @export syntax for more control over output formats (e.g., specifying the delimiter for CSV-like outputs)

Notable bug fixes and internal improvements

• New dictionary implementation significantly reduces memory consumption on large datasets

Current Functionality

• Execution (materialization) of Datalog extended with stratified negation and existential rules (tuple-generating dependencies)
• Loading input data from local or remote CSV/TSV and RDF (NT, Turtle, RDF/XML, NQ, TriG) files
• Writing results to CSV files
• Compatibility with RDF and SPARQL syntax for IRIs and literals
• Datatypes integer (whole numbers), float64 (64bit floating point numbers), string (Unicode strings), and any (union type that can represent any element; default for most contexts)
• Support for built-in predicates <=, <, >=, >, !=, and = and built-in functions +, -, *, and / for types integer and float64
• Aggregation: #count, #min, #max, and #sum

Nemo v0.4.0 is built for mid-sized computing tasks that can still be processed on a normal laptop in seconds or minutes (typically hundreds of thousands to hundreds of millions of facts). In such cases, Nemo is already quite fast – at least fast enough to outperform existing free rule engines on the tests we conducted so far. Example tasks and benchmark results can be found in our sister repository Nemo examples and benchmarks.

Note that the combination of existential quantifiers and stratified negation do not have a standard semantics, and may lead to unexpected conclusions which can be avoided by careful modeling (see Ellmauthaler, Krötzsch, Mennicke; AAAI 2022).

Known Limitations

The following features are not included in v0.4.0 yet and will be added in upcoming releases:

• Support for RDF output
• Support for unary - in arithmetic expressions (current workaround: use 0-?X instead of -?X)
• More built-in functions, especially for string data
• More control over output formats

Moreover, the documentation, though improved, is still very limited. This will be expanded gradually.

Full Changelog: v0.3.0...v0.4.0

Nemo v0.3.0

Version 0.3.0 of Nemo extends the fast in-memory rule reasoner with support for more arithmetic features and extended data sources and formats. The command-line client nmo can be used to access this functionality (use nmo --help for a brief documentation). The online documentation of Nemo covers many of the current features.

New Features and Improvements

The following new features have been added in Nemo v0.3.0:

• Support for arithmetic operators (+, -, *, /), so far only in rule heads
• Extended arithmetic comparisons between variables (can now include two variables, not just variables and constants)
• New datatype string to load plain string data (generally faster loading than the full RDF term parsing of type any, which also can hold string data, as before)
• Support for online sources: just use URLs (with http:/https:) in @source declarations
• Support for RDF files in Turtle and RDF/XML format through the existing load-rdf method (any @base declaration from the rules file will be used as default base IRI in such formats), enabled by Oxigraph's Rio RDF library

Notable bug fixes and internal improvements:

• @output (the directive that specifies which predicate data to return) is now allowed in any place of the file (not just as the end)

Current Functionality

At this version, Nemo therefore includes support for the following features:

• Execution (materialization) of Datalog extended with stratified negation and existential rules (tuple-generating dependencies)
• Loading input data from local or remote CSV/TSV and RDF (NT, Turtle, RDF/XML) files
• Writing results to CSV files
• Compatibility with RDF and SPARQL syntax for IRIs and literals
• Datatypes integer (whole numbers), float64 (64bit floating point numbers), string (Unicode strings), and any (union type that can represent any element; default for most contexts)
• Support for built-in predicates <=, <, >=, >, and = and built-in functions +, -, *, and / (in rule heads) for types integer and float64

Nemo v0.3.0 is built for mid-sized computing tasks that can still be processed on a normal laptop in seconds or minutes (typically hundreds of thousands to hundreds of millions of facts). In such cases, Nemo is already quite fast – at least fast enough to outperform existing free rule engines on the tests we conducted so far. Example tasks and benchmark results can be found in our sister repository Nemo examples and benchmarks.

Note that the combination of existential quantifiers and stratified negation do not have a standard semantics, and may lead to unexpected conclusions which can be avoided by careful modeling (see Ellmauthaler, Krötzsch, Mennicke; AAAI 2022).

Known Limitations

The following features are not included in v0.3.0 yet and will be added in upcoming releases:

• Support for RDF output
• Support for arithmetic functions in rule bodies (currently only in heads)
• Support for unary - in arithmetic expressions (current workaround: use 0-?X instead of -?X)
• More built-in functions, especially for string data, and support for aggregates
• More control over output formats

Moreover, the documentation, though improved, is still very limited. This will be expanded gradually.

Breaking change: CSV default type

The default type for CSV data sources has changed from any to string. For existing rules files, this may require small changes in rules or in CSV files. For example, using type any as before, a quoted RDF string literal would be encoded in CSV as """Hello world!""", where the outer " delimit the value, and the inner "" encode single quotes (in CSV). The result is a string with the characters Hello world!. When using type string instead, the whole value found in CSV will be considered the contents of a string, so the above fragment would produce the string with content "Hello world!" (with the quotes as first and last symbol). To get the same as before, one would leave away the inner "" in CSV, which were only needed to mark the value as string literal in the context of RDF (where many other types of terms are allowed).

One can also revert to the old behaviour by declaring type any explicitly for CSV sources, but this will in general be slower (RDF term parsing and normalization is more work than plain string reading). Finally, one could also read a CSV file that contains RDF-formatted data as strings and work with the plain strings throughout the program. This might require changes to constant that appear in the rules file (e.g., if a former string constant "Hello world!" needs to become "\"Hello world!\"" to match the verbatim data when read as string. Note that this disables all normalization, e.g., "Hello world!" will not be equal to "Hello world!"^^<http://www.w3.org/2001/XMLSchema#string> as it would be in RDF.

Feedback and issue reports are welcome!

Nemo v0.2.0

Version 0.2.0 of Nemo extends the fast in-memory rule reasoner with support for negation and RDF inputs. The command-line client nmo can be used to access this functionality (use nmo --help for a brief documentation). Moreover, the online documentation of Nemo has been expanded to cover many of the current features.

New Features and Improvements

The following new features have been added in Nemo v0.2.0:

• Support for stratified negation: the ~ symbol can be used as a negation operator in rule bodies
• Support for reading RDF data from N-Triples files, as in this example:
  @source triples[3]: load-rdf("sources/example2.nt") . % load into predicate "triples"; can also read .nt.gz

Important bug fixes and internal improvements:

• The type integer now supports arbitrary signed 64bit integers (rather than just unsigned values)
• Much improved result materialization, using a reworked streaming iterator for query results
• Linux binaries are built in a better way that leads to better performance (Issue 263)

Current Functionality

At this version, Nemo therefore includes support for the following features:

• Execution (materialization) of Datalog extended with stratified negation and existential rules (tuple-generating dependencies)
• Loading input data from CSV/TSV and RDF (NT) files, and writing results to CSV files
• Compatibility with RDF and SPARQL syntax for IRIs and literals
• Datatypes integer (whole numbers), float64 (64bit floating point numbers), and any (union type that can represent any element; default)
• Support for built-ins <=, <, >=, and > for type integer

Nemo v0.2.0 is built for mid-sized computing tasks that can still be processed on a normal laptop in seconds or minutes (typically hundreds of thousands to hundreds of millions of facts). In such cases, Nemo is already quite fast – at least fast enough to outperform existing free rule engines on all tests we conducted so far. Example tasks and benchmark results can be found in our sister repository Nemo examples and benchmarks.

Note that the combination of existential quantifiers and stratified negation do not have a standard semantics, and may lead to unexpected conclusions which can be avoided by careful modeling (see Ellmauthaler, Krötzsch, Mennicke; AAAI 2022).

Known Limitations

The following features are not included in v0.2.0 yet and will be added in upcoming releases:

• Support for RDF output
• More built-ins for datatypes, and more datatypes (note that already now Nemo can represent all kinds of values in type any, but it will be convenient to restrict to special cases such as string)
• More built-in functions, including aggregates

Moreover, the documentation, though improved, is still very limited. This will be expanded gradually.

Feedback and issue reports are welcome!

Nemo v0.1.0

Version 0.1.0 of Nemo already provides core features that make it usable as a fast in-memory rule reasoner. The command-line client nmo can be used to access this functionality (use nmo --help for a brief documentation).

New Features

At this version, Nemo includes support for the following features:

• Execution (materialization) of Datalog extended with existential rules (tuple-generating dependencies)
• Loading input data from CSV and TSV files, and writing results to CSV files
• Compatibility with RDF and SPARQL syntax for IRIs and literals
• Datatypes integer (whole numbers), float64 (64bit floating point numbers), and any (union type that can represent any element; default)
• Support for built-ins <=, <, >=, and > for type integer

Nemo v0.1.0 is built for mid-sized computing tasks that can still be processed on a normal laptop in seconds or minutes (typically hundreds of thousands to hundreds of millions of facts). In such cases, Nemo is already quite fast – at least fast enough to outperform existing free rule engines on all tests we conducted so far. Example tasks and benchmark results can be found in our sister repository Nemo examples and benchmarks.

Known Limitations

There is a known problem with the binaries that were automatically built for this release. For best performance, please do not use the attached files but build Nemo on your machine using Rust (command cargo build -r from the nemo directory).

The following features are not included in v0.1.0 yet and will be added in the next release:

• Stratified negation
• Support for RDF input and output
• More built-ins for datatypes, and more datatypes (note that already now Nemo can represent all kinds of values in type any, but it will be convenient to restrict to special cases such as string)

Moreover, the documentation is still very limited. This will be expanded gradually.

Feedback and issue reports are welcome!