From 9d0e07f4f0ce6d97b8baafc8901af4315f4d2213 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Mon, 18 May 2020 16:30:34 +0200 Subject: [PATCH 01/13] [tutorial] Add validation introduction/example --- doc/tutorial.rst | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/doc/tutorial.rst b/doc/tutorial.rst index a50d8d97..9b939e42 100644 --- a/doc/tutorial.rst +++ b/doc/tutorial.rst @@ -983,6 +983,49 @@ A cardinality is set via its convenience method: >>> # or >>> prop.val_cardinality = None +Working with Validations +------------------------ + +odML Validations are a set of pre-defined checks that are run against an odML document automatically when it is saved or loaded. A document cannot be saved, if a Validation fails a check that is classified as an Error. Most validation checks are Warnings that are supposed to raise the overall data quality of the odml Document. + +When an odML document is saved or loaded, tha automatic validation will print a short report of encountered Validation Warnings and it is up to the user whether they want to resolve the Warnings. The odML document provides the ``validate`` method to gain easy access to the default validations. A Validation in turn provides not only a specific description of all encountered warnings or errors within an odML document, but it also provides direct access to each and every odML entity i.e. an odml.Section or an odml.Property where am issue has been found. This enables the user to quickly access and fix an encountered issue. + +A minimal example shows how a workflow using default validations might look like: + + >>> # Create a minimal document with Section issues: name and type are not assigned + >>> doc = odml.Document() + >>> sec = odml.Section(parent=doc) + >>> odml.save(doc, "validation_example.odml.xml") + +This minimal example document will be saved, but will also print the following Validation report: + + >>> UserWarning: The saved Document contains unresolved issues. Run the Documents 'validate' method to access them. + >>> Validation found 0 errors and 2 warnings in 1 Sections and 0 Properties. + +To fix the encountered warnings, users can access the validation via the documents' ``validate`` method: + + >>> validation = doc.validate() + >>> for issue in validation.errors: + >>> print(issue) + +This will show that the validation has encountered two Warnings and also displays the offending odml entity. + + >>> ValidationWarning: Section[73f29acd-16ae-47af-afc7-371d57898e28] 'Section type not specified' + >>> ValidationWarning: Section[73f29acd-16ae-47af-afc7-371d57898e28] 'Name not assigned' + +To fix the "Name not assigned" warning the Section can be accessed via the validation entry and used to directly assign a human readable name to Section in the original document. Re-running the validation will show, that the warning has been removed. + + >>> validation.errors[1].obj.name = "validation_example_section" + >>> # Check that the section name has been changed in the document + >>> print(doc.sections) + >>> # Re-running validation + >>> validation = doc.validate() + >>> for issue in validation.errors: + >>> print(issue) + +Similarly the second validation warning can be resolved before saving the document again. + +Please note that the automatic validation is run whenever a document is saved or loaded using the ``odml.save`` and ``odml.load`` functions as well as the ``ODMLWriter`` or the ``ODMLReader``. The validation is not run when using any of the lower level ``xmlparser``, ``dict_parser`` or ``rdf_converter`` classes. Advanced knowledge on Values ---------------------------- From 4bbe5378e694b9a01b6d101e47f9f03c92d73619 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Mon, 18 May 2020 16:31:00 +0200 Subject: [PATCH 02/13] [tutorial] Add list of default validations --- doc/tutorial.rst | 80 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/doc/tutorial.rst b/doc/tutorial.rst index 9b939e42..471dae72 100644 --- a/doc/tutorial.rst +++ b/doc/tutorial.rst @@ -1027,6 +1027,86 @@ Similarly the second validation warning can be resolved before saving the docume Please note that the automatic validation is run whenever a document is saved or loaded using the ``odml.save`` and ``odml.load`` functions as well as the ``ODMLWriter`` or the ``ODMLReader``. The validation is not run when using any of the lower level ``xmlparser``, ``dict_parser`` or ``rdf_converter`` classes. +List of available default validations +************************************* + +The following contains a list of the default odml validations, their message and the suggested course of action to resolve the issue. + +| Validation: ``object_required_attributes`` +| Message: "Missing required attribute 'xyz'" +| Applies to: ``Document``, ``Section``, ``Property`` +| Course of action: Add an appropriate value to attribute 'xyz' for the reported odml entity. + +| Validation: ``section_type_must_be_defined`` +| Message: "Section type not specified" +| Applies to: ``Section`` +| Course of action: Fill in the ``type`` attribute of the reported Section. + +| Validation: ``section_unique_ids`` +| Message: "Duplicate id in Section 'secA' and 'secB'" +| Applies to: ``Section`` +| Course of action: IDs have to be unique and a duplicate id was found. Assign a new id for the reported Section. + +| Validation: ``property_unique_ids`` +| Message: "Duplicate id in Property 'propA' and 'propB'" +| Applies to: ``Property`` +| Course of action: IDs have to be unique and a duplicate id was found. Assign a new id for the reported Property + +| Validation: ``section_unique_name_type`` +| Message: "name/type combination must be unique" +| Applies to: ``Section`` +| Course of action: The combination of Section.name and Section.type has to be unique on the same level. Change either name or type of the reported Section. + +| Validation: ``object_unique_name`` +| Message: "Object names must be unique" +| Applies to: ``Document``, ``Section``, ``Property`` +| Course of action: Property name has to be unique on the same level. Change the name of the reported Property. + +| Validation: ``object_name_readable`` +| Message: "Name not assigned" +| Applies to: ``Section``, ``Property`` +| Course of action: When Section or Property names are left empty on creation or set to None, they are automatically assigned the entities uuid. Assign a human readable name to the reported entity. + +| Validation: ``property_terminology_check`` +| Message: "Property 'prop' not found in terminology" +| Applies to: ``Property`` +| Course of action: The reported entity is linked to an repository but the repository is not available. Check if the linked content has moved. + +| Validation: ``property_dependency_check`` +| Message: "Property refers to a non-existent dependency object" or "Dependency-value is not equal to value of the property's dependency" +| Applies to: ``Property`` +| Course of action: The reported entity depends on another Property, but this dependency has not been satisfied. Check the referenced Property and its value to resolve the issue. + +| Validation: ``property_values_check`` +| Message: "Tuple of length 'x' not consistent with dtype 'dtype'!" or "Property values not of consistent dtype!". +| Applies to: ``Property`` +| Course of action: Adjust the values or the dtype of the referenced Propery. + +| Validation: ``property_values_string_check`` +| Message: "Dtype of property "prop" currently is "string", but might fit dtype "dtype"!" +| Applies to: ``Property`` +| Course of action: Check if the datatype of the referenced Property.values has been loaded correctly and change the Property.dtype if required. + +| Validation: ``section_properties_cardinality`` +| Message: "cardinality violated x values, y found)" +| Applies to: ``Section`` +| Course of action: A cardinality defined for the number of Properties of a Section does not match. Add or remove Properties until the cardinality has been satisfied or adjust the cardinality. + +| Validation: ``section_sections_cardinality`` +| Message: "cardinality violated x values, y found)" +| Applies to: ``Section`` +| Course of action: A cardinality defined for the number of Sections of a Section does not match. Add or remove Sections until the cardinality has been satisfied or adjust the cardinality. + +| Validation: ``property_values_cardinality`` +| Message: "cardinality violated x values, y found)" +| Applies to: ``Property`` +| Course of action: A cardinality defined for the number of Values of a Property does not match. Add or remove Values until the cardinality has been satisfied or adjust the cardinality. + +| Validation: ``section_repository_present`` +| Message: "A section should have an associated repository" or "Could not load terminology" or "Section type not found in terminology" +| Applies to: ``Section`` +| Course of action: Optional validation. Will report any section that does not specify a repository. Add a repository to the reported Section to resolve. + Advanced knowledge on Values ---------------------------- From b6d6ce0bead6d32beaea37df2af23e94f92a6073 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Mon, 18 May 2020 17:03:35 +0200 Subject: [PATCH 03/13] [validation] Add custom validation IssueID --- odml/validation.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/odml/validation.py b/odml/validation.py index 800fc793..34162d4a 100644 --- a/odml/validation.py +++ b/odml/validation.py @@ -51,6 +51,9 @@ class IssueID(Enum): # Optional validations section_repository_present = 600 + # Custom validation + custom_validation = 701 + class ValidationError(object): """ From a0b2d0df6a66ed0224ca202f4e6a6525d35c3b1a Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Mon, 18 May 2020 17:03:58 +0200 Subject: [PATCH 04/13] [tutorial] Add custom validation section --- doc/tutorial.rst | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/doc/tutorial.rst b/doc/tutorial.rst index 471dae72..6808a1c0 100644 --- a/doc/tutorial.rst +++ b/doc/tutorial.rst @@ -1107,6 +1107,40 @@ The following contains a list of the default odml validations, their message and | Applies to: ``Section`` | Course of action: Optional validation. Will report any section that does not specify a repository. Add a repository to the reported Section to resolve. +Custom validations +****************** + +Users can write their own validation and register them either with the default validation or add it to their own validation class instance. + +A custom validation handler needs to ``yield`` a ``ValidationError``. See the ``validation.ValidationError`` class for details. + +Custom validation handlers can be registered to be applied on ``odML`` (the odml Document), ``section`` or ``property``. + + >>> import odml + >>> import odml.validation as oval + >>> + >>> # Create an example document + >>> doc = odml.Document() + >>> sec_valid = odml.Section(name="Recording-20200505", parent=doc) + >>> sec_invalid = odml.Section(name="Movie-20200505", parent=doc) + >>> subsec = odml.Section(name="Sub-Movie-20200505", parent=sec_valid) + >>> + >>> # Define a validation handler that yields a ValidationError if a section name does not start with 'Recording-' + >>> def custom_validation_handler(obj): + >>> validation_id = oval.IssueID.custom_validation + >>> msg = "Section name does not start with 'Recording-'" + >>> if not obj.name.startswith("Recording-"): + >>> yield oval.ValidationError(obj, msg, oval.LABEL_ERROR, validation_id) + >>> + >>> # Create a custom, empty validation with an odML document 'doc' + >>> custom_validation = oval.Validation(doc, reset=True) + >>> # Register a custom validation handler that should be applied on all Sections of a Document + >>> custom_validation.register_custom_handler("section", custom_validation_handler) + >>> # Run the custom validation and return a report + >>> custom_validation.report() + >>> # Display the errors reported by the validation + >>> print(custom_validation.errors) + Advanced knowledge on Values ---------------------------- From 603f866927f6496438f6f7968fcc218777a22f14 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Tue, 19 May 2020 09:14:16 +0200 Subject: [PATCH 05/13] [README] Add Python 2 support removal note --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 57f7cf41..4ea3455e 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ release notes](https://github.com/G-Node/python-odml/releases). ## Previous Python versions -Python 2 has reached end of life. We will not keep any future versions of odml Python 2 compatible and also recommend using a Python version >= 3.6. If a Python version < 3.6 is a requirement, the following dependency needs to be installed as well: +Python 2 has reached end of life. We will not keep any future versions of odml Python 2 compatible and will completely drop support for Python 2 with August 2020. We also recommend using a Python version >= 3.6. If a Python version < 3.6 is a requirement, the following dependency needs to be installed as well: * pip install * enum34 (version 0.4.4) From f7dbc2229bfb78ba25f0c6aa669bb90ed5d30e53 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Tue, 19 May 2020 11:36:43 +0200 Subject: [PATCH 06/13] [CHANGELOG] Add 1.5.0 release notes --- CHANGELOG.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e55196b2..2e623cca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,66 @@ Used to document all changes from previous releases and collect changes until the next release. +# Version 1.5.0 + +# Python 2 deprecation warning +A Python 2 deprecation warning for August 2020 has been added. See issue #387 for details. + +# Validation feature update +See issues #377, #378 and #379 as well as Pull Request #389 for details. + +An `IssueID` enum class as been added to provide identifiers to individual ValidationErrors. The `Validation` class itself has been refactored to provide the option to create standalone Validation instances with a different set of registered validations than the default library validation. +The `Validation` class now features the new `register_custom_handler`, `run_validation`and `report` methods to add custom validation handlers to an instance, re-run the validations of an existing Instance and provide a brief report of encountered errors and warnings. The general `ValidationError.__repr__` string has been shortened to make the individual ValidationErrors more convenient to print and read. The default Validation is always run when a Document is saved or loaded via the `ODMLParser` and the `Validation.report` method is used to provide a `warnings.warn` message of the following format: +``` +UserWarning: The saved Document contains formal issues. Run 'odml.validation.Validation(doc)' to resolve them. +Validation found 0 errors and 3 warnings in 1 Sections and 1 Properties. +``` + +Further changes to the Validation class and behavior include: +- an odml `Document` now provides a `validate` method that will run a default Validation and return the Validation instance to provide users with access to encountered issues. +- a `validation_id` field has been added to the `ValidationError` class. +- standalone Sections and Properties can now be validated. +- Sections and Properties are validated on init. +- the `section_repository_present` validation has been removed from the default validation list. Since Sections rarely have repositories set, this validation can lead to spam when validating a Document. + +# Cardinality feature +Property and Section now provide a cardinality feature. Users can now define a range how many Values a Property and how many Properties or Sections a Section should have. A cardinality can be set and read via its accessor method and can be set via an additional convenience method. Whenever a cardinality or an affected Value, Section or Property is set, a corresponding validation is triggered. If this a set cardinality for a Property or Section is violated, a message is printed to the command line directly and a warning is issued when a Document is saved or loaded. Every cardinality is saved to and loaded from all available file formats. +The full functionality of all cardinality features are documented in the tutorial and is available via readthedocs. For additional details see pull requests #374, #382, #383, #384 and issue #361. + +# Update in Section type default behavior +With recent updates the library now respects and enforces `Section.type` as a required attribute and allows save only with documents where this requirement is satisfied. +To allow backwards file compatibility and ease usage, `Section.type` is by default set to the string `n.s.` (not specified), which means files where no `Section.type` had been specified can be loaded and saved, but will contain `n.s.` as value for every `Sections.type` that was previously not specified. +Further the validation run before a document can be saved will issue a warning, if a `Section.type` with value `n.s.` is encountered and will still refuse to save with an error, if an empty `Section.type` is encountered. See PR #376 for details. + +# DictParser and ODMLParser fully support ignore errors +- the `DictParser.DictReader` is now able to ignore errors, collect warnings and print corresponding notifications and works now analogous to the `xmlparser.XMLReader` behaviour. See issue #367 for details. +- the `ODMLParser.ODMLReader` for JSON and YAML now uses `ignore_errors` by default e.g. when using the `odml.load` function for JSON and YAML odml files. + +# Fixes +- fixes an exception when trying to append or extend a `Property` with dtype `tuple`. See issue #364 for details. +- when trying to set the `name` attribute to `None`, it now silently sets the name to `id` instead, since `name` must not be empty. It would be set to `id` on load and can cause `AttributeError` exceptions with some methods if its not set. +- a bug was fixed in `format.revmap` where the reverse mapping of an odml attribute would always return the case that the attribute is part of the format, even if it was not. + +# Minor changes and updates +- all deprecation warnings now use the warnings module. +- the `Property.value` attribute deprecation warnings have been unified. See issue #360 for details. +- the `base.Sectionable.create_section` method has been updated to conform with `Section.__init__`. See issue #368 for details. +- all saved XML odML files now use the same XML header. See issue #339 for details. +- a function to manually refresh the terminology cache has been added. See issue #202 for details. +- a Validation to note non-human readable `Property` and `Section` names has been added. See issue #365 for details. +- getter and setter methods are added to the `odml.Document.origin_file_name` attribute. See issue #358 for details. +- the Exception type in `odml.tools.converters.VersionConverter` is changed to `odml.tools.parser_utils.ParserException`. See issue #359 for details. +- the `odml.Property.export_leaf` method now also includes sibling Properties on export. +- the `rdf_converter` has been cleaned up, see issues #211 and #345 for details. +- the test for the `Section`/`Property` order in documents obtained via the `RDFReader` has been expanded. See issue #265 for details. +- tests for Validation errors on `Section` or `Property` init have been added. See issue #369 for details. +- tests writing temporary files now properly clean up after themselves. See issue #381 for details. +- tests now use a common temporary directory to write files and use a constant for accessing the test/resources directory. +- the link to the odML tutorial in the README file now points to python-odml.readthedocs.org; the README file now also includes links to Travis and Coveralls. +- the tutorial now includes descriptions of the `pprint` method and a link to the odML templates hosting site. Further the tutorial has been updated to include descriptions of the cardinality feature and Validation usage. +- introduces major PEP8 fixes to basically all files of the library. See Pull Request #385 for details. +- the class reference now includes the Template, Terminology and Validation classes. + # Version 1.4.5 ## Minor changes, updates and fixes. From 8ee5bb8cf7de684b4b6f776c80665fbf61365fd6 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Tue, 19 May 2020 11:41:20 +0200 Subject: [PATCH 07/13] [doc] Add support-classes.rst --- doc/reference.rst | 1 + doc/support-classes.rst | 42 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) create mode 100644 doc/support-classes.rst diff --git a/doc/reference.rst b/doc/reference.rst index 078d4f46..165b51aa 100644 --- a/doc/reference.rst +++ b/doc/reference.rst @@ -6,4 +6,5 @@ Class-Reference :maxdepth: 2 base-classes + support-classes tools diff --git a/doc/support-classes.rst b/doc/support-classes.rst new file mode 100644 index 00000000..8a31e539 --- /dev/null +++ b/doc/support-classes.rst @@ -0,0 +1,42 @@ +.. _support_classes: + +odML-Support Classes +==================== + +These classes are + + +Validation +------- +.. autoclass:: odml.validation.Validation + :members: + :inherited-members: + :undoc-members: + +IssueID +------- +.. autoclass:: odml.validation.IssueID + :members: + :inherited-members: + :undoc-members: + +ValidationError +--------------- +.. autoclass:: odml.validation.ValidationError + :members: + :inherited-members: + :undoc-members: + +TemplateHandler +--------------- +.. autoclass:: odml.templates.TemplateHandler + :members: + :inherited-members: + :undoc-members: + +Terminologies +------------- +.. autoclass:: odml.terminology.Terminologies + :members: + :inherited-members: + :undoc-members: From eaf285295314d0dee1ee8e51984c40781e83b3f2 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Tue, 19 May 2020 12:46:35 +0200 Subject: [PATCH 08/13] [tutorial] Add cardinality note --- doc/tutorial.rst | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/doc/tutorial.rst b/doc/tutorial.rst index 6808a1c0..e18454b7 100644 --- a/doc/tutorial.rst +++ b/doc/tutorial.rst @@ -983,12 +983,14 @@ A cardinality is set via its convenience method: >>> # or >>> prop.val_cardinality = None +Please note that a set cardinality is not enforced. Users can set less or more entities than are specified allowed via a cardinality. Instead whenever a cardinality is not met, a warning message is displayed and any unment cardinality will show up as a Validation warning message whenever a document is saved or loaded. + Working with Validations ------------------------ odML Validations are a set of pre-defined checks that are run against an odML document automatically when it is saved or loaded. A document cannot be saved, if a Validation fails a check that is classified as an Error. Most validation checks are Warnings that are supposed to raise the overall data quality of the odml Document. -When an odML document is saved or loaded, tha automatic validation will print a short report of encountered Validation Warnings and it is up to the user whether they want to resolve the Warnings. The odML document provides the ``validate`` method to gain easy access to the default validations. A Validation in turn provides not only a specific description of all encountered warnings or errors within an odML document, but it also provides direct access to each and every odML entity i.e. an odml.Section or an odml.Property where am issue has been found. This enables the user to quickly access and fix an encountered issue. +When an odML document is saved or loaded, tha automatic validation will print a short report of encountered Validation Warnings and it is up to the user whether they want to resolve the Warnings. The odML document provides the ``validate`` method to gain easy access to the default validations. A Validation in turn provides not only a specific description of all encountered warnings or errors within an odML document, but it also provides direct access to each and every odML entity i.e. an ``odml.Section`` or an ``odml.Property`` where an issue has been found. This enables the user to quickly access and fix an encountered issue. A minimal example shows how a workflow using default validations might look like: @@ -1013,7 +1015,7 @@ This will show that the validation has encountered two Warnings and also display >>> ValidationWarning: Section[73f29acd-16ae-47af-afc7-371d57898e28] 'Section type not specified' >>> ValidationWarning: Section[73f29acd-16ae-47af-afc7-371d57898e28] 'Name not assigned' -To fix the "Name not assigned" warning the Section can be accessed via the validation entry and used to directly assign a human readable name to Section in the original document. Re-running the validation will show, that the warning has been removed. +To fix the "Name not assigned" warning the Section can be accessed via the validation entry and used to directly assign a human readable name to the Section in the original document. Re-running the validation will show, that the warning has been removed. >>> validation.errors[1].obj.name = "validation_example_section" >>> # Check that the section name has been changed in the document @@ -1025,7 +1027,7 @@ To fix the "Name not assigned" warning the Section can be accessed via the valid Similarly the second validation warning can be resolved before saving the document again. -Please note that the automatic validation is run whenever a document is saved or loaded using the ``odml.save`` and ``odml.load`` functions as well as the ``ODMLWriter`` or the ``ODMLReader``. The validation is not run when using any of the lower level ``xmlparser``, ``dict_parser`` or ``rdf_converter`` classes. +Please note that the automatic validation is run whenever a document is saved or loaded using the ``odml.save`` and ``odml.load`` functions as well as the ``ODMLWriter`` or the ``ODMLReader`` class. The validation is not run when using any of the lower level ``xmlparser``, ``dict_parser`` or ``rdf_converter`` classes. List of available default validations ************************************* @@ -1114,7 +1116,7 @@ Users can write their own validation and register them either with the default v A custom validation handler needs to ``yield`` a ``ValidationError``. See the ``validation.ValidationError`` class for details. -Custom validation handlers can be registered to be applied on ``odML`` (the odml Document), ``section`` or ``property``. +Custom validation handlers can be registered to be applied on "odML" (the odml Document), "section" or "property". >>> import odml >>> import odml.validation as oval @@ -1141,8 +1143,8 @@ Custom validation handlers can be registered to be applied on ``odML`` (the odml >>> # Display the errors reported by the validation >>> print(custom_validation.errors) -Advanced knowledge on Values ----------------------------- +Advanced Value features +----------------------- Data type conversions ********************* @@ -1178,8 +1180,6 @@ converted to integer and then back to float:: Links & Includes **************** -Please note, that this section is outdated but still valid. - Sections can be linked to other Sections, so that they include their defined attributes. A link can be within the document (``link`` property) or to an external one (``include`` property). @@ -1211,8 +1211,6 @@ then set merge with the new object. Terminologies ************* -Please note, that this section is outdated but still valid. - odML supports terminologies that are data structure templates for typical use cases. Sections can have a ``repository`` attribute. As repositories can be inherited, the current applicable one can be obtained using the From d7acb4abec087736b1078a05faeadb7a5939d8bd Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Tue, 19 May 2020 15:39:47 +0200 Subject: [PATCH 09/13] [validation] Remove default terminology check The property_terminology_check validation can lead to a threading lock in some cases when using Python 3.8 though I was not able to figure out the details yet. Therefore the validation is kept, but removed from the default Validation checks until this can be resolved. Also brings the docstring up to date and adjusts the corresponding test. --- odml/validation.py | 7 +------ test/test_validation.py | 3 +++ 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/odml/validation.py b/odml/validation.py index 34162d4a..4ad8c872 100644 --- a/odml/validation.py +++ b/odml/validation.py @@ -451,9 +451,7 @@ def object_name_readable(obj): def property_terminology_check(prop): """ - 1. warn, if there are properties that do not occur in the terminology. - 2. warn, if there are multiple values with different units or the unit does - not match the one in the terminology. + Tests if there are properties that do not occur in the terminology. """ validation_id = IssueID.property_terminology_check @@ -470,9 +468,6 @@ def property_terminology_check(prop): yield ValidationError(prop, msg, LABEL_WARNING, validation_id) -Validation.register_handler('property', property_terminology_check) - - def property_dependency_check(prop): """ Produces a warning if the dependency attribute refers to a non-existent attribute diff --git a/test/test_validation.py b/test/test_validation.py index a03b1725..0a5dc6f1 100644 --- a/test/test_validation.py +++ b/test/test_validation.py @@ -208,6 +208,9 @@ def test_uniques(self): """) def test_property_in_terminology(self): + odml.validation.Validation.register_handler("property", + odml.validation.property_terminology_check) + doc = samplefile.parse(""" s1[t1] - P1 From 5e4deed593567c5ccdd0989ea5372d2617333d56 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Tue, 19 May 2020 17:16:51 +0200 Subject: [PATCH 10/13] [validation] Exit None Property.value value checks The property value validations will break if a Property values list contains a 'None' value. As a failsafe exit these validations if a 'None' value is encountered. --- odml/validation.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/odml/validation.py b/odml/validation.py index 4ad8c872..2a74b199 100644 --- a/odml/validation.py +++ b/odml/validation.py @@ -514,6 +514,10 @@ def property_values_check(prop): return for val in prop.values: + # Do not continue if a value is None + if val is None: + return + if dtype.endswith("-tuple"): tuple_len = int(dtype[:-6]) if len(val) != tuple_len: @@ -556,6 +560,10 @@ def property_values_string_check(prop): val_dtypes = [] for val in prop.values: + # Do not continue if a value is None + if val is None: + return + curr_dtype = "string" for check_dtype in dtype_checks.items(): From eab8e6ae3bfe31aee18df683f0e4e37ae5dc7a74 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Tue, 19 May 2020 17:31:17 +0200 Subject: [PATCH 11/13] [section] Use custom cardinality checks validation The cardinality checks are run quite frequently. To reduce the load of background checks on the Document, custom validations that only run the specific cardinality checks are used instead. --- odml/section.py | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/odml/section.py b/odml/section.py index 3a015978..83b23ef2 100644 --- a/odml/section.py +++ b/odml/section.py @@ -420,7 +420,12 @@ def _sections_cardinality_validation(self): Runs a validation to check whether the sections cardinality is respected and prints a warning message otherwise. """ - valid = validation.Validation(self) + # This check is run quite frequently so do not run all checks via the default validation + # but use a custom validation instead. + valid = validation.Validation(self, validate=False, reset=True) + valid.register_custom_handler("section", validation.section_sections_cardinality) + valid.run_validation() + val_id = validation.IssueID.section_sections_cardinality # Make sure to display only warnings of the current section @@ -475,7 +480,12 @@ def _properties_cardinality_validation(self): Runs a validation to check whether the properties cardinality is respected and prints a warning message otherwise. """ - valid = validation.Validation(self) + # This check is run quite frequently so do not run all checks via the default validation + # but use a custom validation instead. + valid = validation.Validation(self, validate=False, reset=True) + valid.register_custom_handler("section", validation.section_properties_cardinality) + valid.run_validation() + val_id = validation.IssueID.section_properties_cardinality # Make sure to display only warnings of the current section From e8df73568d0dd03026c16cf04ce188383b0f9ec2 Mon Sep 17 00:00:00 2001 From: "M. Sonntag" Date: Tue, 19 May 2020 17:33:36 +0200 Subject: [PATCH 12/13] [property] Use custom cardinality check validation The cardinality checks are run quite frequently; everytime a value is changed in addition to changes of a cardinality itself. To reduce the load of background checks on the Document, a custom validations that only runs the specific value cardinality check is used instead. --- odml/property.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/odml/property.py b/odml/property.py index 82537bda..4526045a 100644 --- a/odml/property.py +++ b/odml/property.py @@ -573,7 +573,12 @@ def _values_cardinality_validation(self): Runs a validation to check whether the values cardinality is respected and prints a warning message otherwise. """ - valid = validation.Validation(self) + # This check is run quite frequently so do not run all checks via the default validation + # but use a custom validation instead. + valid = validation.Validation(self, validate=False, reset=True) + valid.register_custom_handler("property", validation.property_values_cardinality) + valid.run_validation() + val_id = validation.IssueID.property_values_cardinality # Make sure to display only warnings of the current property From 9fd5ba208ee1a7f954d76035a339c8b5f4bada2d Mon Sep 17 00:00:00 2001 From: Michael Sonntag Date: Sat, 23 May 2020 14:15:57 +0200 Subject: [PATCH 13/13] Update doc/tutorial.rst Co-authored-by: Achilleas Koutsou --- doc/tutorial.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.rst b/doc/tutorial.rst index e18454b7..096eafdb 100644 --- a/doc/tutorial.rst +++ b/doc/tutorial.rst @@ -1072,7 +1072,7 @@ The following contains a list of the default odml validations, their message and | Validation: ``property_terminology_check`` | Message: "Property 'prop' not found in terminology" | Applies to: ``Property`` -| Course of action: The reported entity is linked to an repository but the repository is not available. Check if the linked content has moved. +| Course of action: The reported entity is linked to a repository but the repository is not available. Check if the linked content has moved. | Validation: ``property_dependency_check`` | Message: "Property refers to a non-existent dependency object" or "Dependency-value is not equal to value of the property's dependency"