Merge pull request #391 from mpsonntag/release

Preparation for version 1.5.0 release

LGTM

CHANGELOG.md

@@ -3,6 +3,66 @@
 Used to document all changes from previous releases and collect changes 
 until the next release.
 
+# Version 1.5.0
+
+# Python 2 deprecation warning
+A Python 2 deprecation warning for August 2020 has been added. See issue #387 for details.
+
+# Validation feature update
+See issues #377, #378 and #379 as well as Pull Request #389 for details.
+
+An `IssueID` enum class as been added to provide identifiers to individual ValidationErrors. The `Validation` class itself has been refactored to provide the option to create standalone Validation instances with a different set of registered validations than the default library validation.
+The `Validation` class now features the new `register_custom_handler`, `run_validation`and `report` methods to add custom validation handlers to an instance, re-run the validations of an existing Instance and provide a brief report of encountered errors and warnings. The general `ValidationError.__repr__` string has been shortened to make the individual ValidationErrors more convenient to print and read. The default Validation is always run when a Document is saved or loaded via the `ODMLParser` and the `Validation.report` method is used to provide a `warnings.warn` message of the following format:
+```
+UserWarning: The saved Document contains formal issues. Run 'odml.validation.Validation(doc)' to resolve them.
+Validation found 0 errors and 3 warnings in 1 Sections and 1 Properties.
+```
+
+Further changes to the Validation class and behavior include:
+- an odml `Document` now provides a `validate` method that will run a default Validation and return the Validation instance to provide users with access to encountered issues.
+- a `validation_id` field has been added to the `ValidationError` class.
+- standalone Sections and Properties can now be validated.
+- Sections and Properties are validated on init.
+- the `section_repository_present` validation has been removed from the default validation list. Since Sections rarely have repositories set, this validation can lead to spam when validating a Document.
+
+# Cardinality feature
+Property and Section now provide a cardinality feature. Users can now define a range how many Values a Property and how many Properties or Sections a Section should have. A cardinality can be set and read via its accessor method and can be set via an additional convenience method. Whenever a cardinality or an affected Value, Section or Property is set, a corresponding validation is triggered. If this a set cardinality for a Property or Section is violated, a message is printed to the command line directly and a warning is issued when a Document is saved or loaded. Every cardinality is saved to and loaded from all available file formats.
+The full functionality of all cardinality features are documented in the tutorial and is available via readthedocs. For additional details see pull requests #374, #382, #383, #384 and issue #361. 
+
+# Update in Section type default behavior
+With recent updates the library now respects and enforces `Section.type` as a required attribute and allows save only with documents where this requirement is satisfied.
+To allow backwards file compatibility and ease usage, `Section.type` is by default set to the string `n.s.` (not specified), which means files where no `Section.type` had been specified can be loaded and saved, but will contain `n.s.` as value for every `Sections.type` that was previously not specified.
+Further the validation run before a document can be saved will issue a warning, if a `Section.type` with value `n.s.` is encountered and will still refuse to save with an error, if an empty `Section.type` is encountered. See PR #376 for details.
+
+# DictParser and ODMLParser fully support ignore errors
+- the `DictParser.DictReader` is now able to ignore errors, collect warnings and print corresponding notifications and works now analogous to the `xmlparser.XMLReader` behaviour. See issue #367 for details.
+- the `ODMLParser.ODMLReader` for JSON and YAML now uses `ignore_errors` by default e.g. when using the `odml.load` function for JSON and YAML odml files.
+
+# Fixes
+- fixes an exception when trying to append or extend a `Property` with dtype `tuple`. See issue #364 for details.
+- when trying to set the `name` attribute to `None`, it now silently sets the name to `id` instead, since `name` must not be empty. It would be set to `id` on load and can cause `AttributeError` exceptions with some methods if its not set.
+- a bug was fixed in `format.revmap` where the reverse mapping of an odml attribute would always return the case that the attribute is part of the format, even if it was not.
+
+# Minor changes and updates
+- all deprecation warnings now use the warnings module.
+- the `Property.value` attribute deprecation warnings have been unified. See issue #360 for details.
+- the `base.Sectionable.create_section` method has been updated to conform with `Section.__init__`. See issue #368 for details.
+- all saved XML odML files now use the same XML header. See issue #339 for details.
+- a function to manually refresh the terminology cache has been added. See issue #202 for details.
+- a Validation to note non-human readable `Property` and `Section` names has been added. See issue #365 for details.
+- getter and setter methods are added to the `odml.Document.origin_file_name` attribute. See issue #358 for details.
+- the Exception type in `odml.tools.converters.VersionConverter` is changed to `odml.tools.parser_utils.ParserException`. See issue #359 for details.
+- the `odml.Property.export_leaf` method now also includes sibling Properties on export.
+- the `rdf_converter` has been cleaned up, see issues #211 and #345 for details.
+- the test for the `Section`/`Property` order in documents obtained via the `RDFReader` has been expanded. See issue #265 for details.
+- tests for Validation errors on `Section` or `Property` init have been added. See issue #369 for details.
+- tests writing temporary files now properly clean up after themselves. See issue #381 for details.
+- tests now use a common temporary directory to write files and use a constant for accessing the test/resources directory.
+- the link to the odML tutorial in the README file now points to python-odml.readthedocs.org; the README file now also includes links to Travis and Coveralls.
+- the tutorial now includes descriptions of the `pprint` method and a link to the odML templates hosting site. Further the tutorial has been updated to include descriptions of the cardinality feature and Validation usage. 
+- introduces major PEP8 fixes to basically all files of the library. See Pull Request #385 for details.
+- the class reference now includes the Template, Terminology and Validation classes.
+
 # Version 1.4.5
 
 ## Minor changes, updates and fixes.

README.md

@@ -93,7 +93,7 @@ release notes](https://github.com/G-Node/python-odml/releases).
 
 ## Previous Python versions
 
-Python 2 has reached end of life. We will not keep any future versions of odml Python 2 compatible and also recommend using a Python version >= 3.6. If a Python version < 3.6 is a requirement, the following dependency needs to be installed as well:
+Python 2 has reached end of life. We will not keep any future versions of odml Python 2 compatible and will completely drop support for Python 2 with August 2020. We also recommend using a Python version >= 3.6. If a Python version < 3.6 is a requirement, the following dependency needs to be installed as well:
 
 * pip install
   * enum34 (version 0.4.4)

doc/reference.rst

@@ -6,4 +6,5 @@ Class-Reference
    :maxdepth: 2
 
    base-classes
+   support-classes
    tools

doc/support-classes.rst

@@ -0,0 +1,42 @@
+.. _support_classes:
+
+odML-Support Classes
+====================
+
+These classes are
+
+
+Validation
+-------
+.. autoclass:: odml.validation.Validation
+   :members:
+   :inherited-members:
+   :undoc-members:
+
+IssueID
+-------
+.. autoclass:: odml.validation.IssueID
+   :members:
+   :inherited-members:
+   :undoc-members:
+
+ValidationError
+---------------
+.. autoclass:: odml.validation.ValidationError
+   :members:
+   :inherited-members:
+   :undoc-members:
+
+TemplateHandler
+---------------
+.. autoclass:: odml.templates.TemplateHandler
+   :members:
+   :inherited-members:
+   :undoc-members:
+
+Terminologies
+-------------
+.. autoclass:: odml.terminology.Terminologies
+   :members:
+   :inherited-members:
+   :undoc-members:

doc/tutorial.rst

@@ -983,9 +983,168 @@ A cardinality is set via its convenience method:
     >>> # or
     >>> prop.val_cardinality = None
 
+Please note that a set cardinality is not enforced. Users can set less or more entities than are specified allowed via a cardinality. Instead whenever a cardinality is not met, a warning message is displayed and any unment cardinality will show up as a Validation warning message whenever a document is saved or loaded.
 
-Advanced knowledge on Values
-----------------------------
+Working with Validations
+------------------------
+
+odML Validations are a set of pre-defined checks that are run against an odML document automatically when it is saved or loaded. A document cannot be saved, if a Validation fails a check that is classified as an Error. Most validation checks are Warnings that are supposed to raise the overall data quality of the odml Document.
+
+When an odML document is saved or loaded, tha automatic validation will print a short report of encountered Validation Warnings and it is up to the user whether they want to resolve the Warnings. The odML document provides the ``validate`` method to gain easy access to the default validations. A Validation in turn provides not only a specific description of all encountered warnings or errors within an odML document, but it also provides direct access to each and every odML entity i.e. an ``odml.Section`` or an ``odml.Property`` where an issue has been found. This enables the user to quickly access and fix an encountered issue.
+
+A minimal example shows how a workflow using default validations might look like:
+
+    >>> # Create a minimal document with Section issues: name and type are not assigned
+    >>> doc = odml.Document()
+    >>> sec = odml.Section(parent=doc)
+    >>> odml.save(doc, "validation_example.odml.xml")
+
+This minimal example document will be saved, but will also print the following Validation report:
+
+    >>> UserWarning: The saved Document contains unresolved issues. Run the Documents 'validate' method to access them.
+    >>> Validation found 0 errors and 2 warnings in 1 Sections and 0 Properties.
+
+To fix the encountered warnings, users can access the validation via the documents' ``validate`` method:
+
+    >>> validation = doc.validate()
+    >>> for issue in validation.errors:
+    >>>     print(issue)
+
+This will show that the validation has encountered two Warnings and also displays the offending odml entity.
+
+    >>> ValidationWarning: Section[73f29acd-16ae-47af-afc7-371d57898e28] 'Section type not specified'
+    >>> ValidationWarning: Section[73f29acd-16ae-47af-afc7-371d57898e28] 'Name not assigned'
+
+To fix the "Name not assigned" warning the Section can be accessed via the validation entry and used to directly assign a human readable name to the Section in the original document. Re-running the validation will show, that the warning has been removed.
+
+    >>> validation.errors[1].obj.name = "validation_example_section"
+    >>> # Check that the section name has been changed in the document
+    >>> print(doc.sections)
+    >>> # Re-running validation
+    >>> validation = doc.validate()
+    >>> for issue in validation.errors:
+    >>>     print(issue)
+
+Similarly the second validation warning can be resolved before saving the document again.
+
+Please note that the automatic validation is run whenever a document is saved or loaded using the ``odml.save`` and ``odml.load`` functions as well as the ``ODMLWriter`` or the ``ODMLReader`` class. The validation is not run when using any of the lower level ``xmlparser``, ``dict_parser`` or ``rdf_converter`` classes.
+
+List of available default validations
+*************************************
+
+The following contains a list of the default odml validations, their message and the suggested course of action to resolve the issue.
+
+| Validation: ``object_required_attributes``
+| Message: "Missing required attribute 'xyz'"
+| Applies to: ``Document``, ``Section``, ``Property``
+| Course of action: Add an appropriate value to attribute 'xyz' for the reported odml entity.
+
+| Validation: ``section_type_must_be_defined``
+| Message: "Section type not specified"
+| Applies to: ``Section``
+| Course of action: Fill in the ``type`` attribute of the reported Section.
+
+| Validation: ``section_unique_ids``
+| Message: "Duplicate id in Section 'secA' and 'secB'"
+| Applies to: ``Section``
+| Course of action: IDs have to be unique and a duplicate id was found. Assign a new id for the reported Section.
+
+| Validation: ``property_unique_ids``
+| Message: "Duplicate id in Property 'propA' and 'propB'"
+| Applies to: ``Property``
+| Course of action: IDs have to be unique and a duplicate id was found. Assign a new id for the reported Property
+
+| Validation: ``section_unique_name_type``
+| Message: "name/type combination must be unique"
+| Applies to: ``Section``
+| Course of action: The combination of Section.name and Section.type has to be unique on the same level. Change either name or type of the reported Section.
+
+| Validation: ``object_unique_name``
+| Message: "Object names must be unique"
+| Applies to: ``Document``, ``Section``, ``Property``
+| Course of action: Property name has to be unique on the same level. Change the name of the reported Property.
+
+| Validation: ``object_name_readable``
+| Message: "Name not assigned"
+| Applies to: ``Section``, ``Property``
+| Course of action: When Section or Property names are left empty on creation or set to None, they are automatically assigned the entities uuid. Assign a human readable name to the reported entity.
+
+| Validation: ``property_terminology_check``
+| Message: "Property 'prop' not found in terminology"
+| Applies to: ``Property``
+| Course of action: The reported entity is linked to a repository but the repository is not available. Check if the linked content has moved.
+
+| Validation: ``property_dependency_check``
+| Message: "Property refers to a non-existent dependency object" or "Dependency-value is not equal to value of the property's dependency"
+| Applies to: ``Property``
+| Course of action: The reported entity depends on another Property, but this dependency has not been satisfied. Check the referenced Property and its value to resolve the issue.
+
+| Validation: ``property_values_check``
+| Message: "Tuple of length 'x' not consistent with dtype 'dtype'!" or "Property values not of consistent dtype!".
+| Applies to: ``Property``
+| Course of action: Adjust the values or the dtype of the referenced Propery.
+
+| Validation: ``property_values_string_check``
+| Message: "Dtype of property "prop" currently is "string", but might fit dtype "dtype"!"
+| Applies to: ``Property``
+| Course of action: Check if the datatype of the referenced Property.values has been loaded correctly and change the Property.dtype if required.
+
+| Validation: ``section_properties_cardinality``
+| Message: "cardinality violated x values, y found)"
+| Applies to: ``Section``
+| Course of action: A cardinality defined for the number of Properties of a Section does not match. Add or remove Properties until the cardinality has been satisfied or adjust the cardinality.
+
+| Validation: ``section_sections_cardinality``
+| Message: "cardinality violated x values, y found)"
+| Applies to: ``Section``
+| Course of action: A cardinality defined for the number of Sections of a Section does not match. Add or remove Sections until the cardinality has been satisfied or adjust the cardinality.
+
+| Validation: ``property_values_cardinality``
+| Message: "cardinality violated x values, y found)"
+| Applies to: ``Property``
+| Course of action: A cardinality defined for the number of Values of a Property does not match. Add or remove Values until the cardinality has been satisfied or adjust the cardinality.
+
+| Validation: ``section_repository_present``
+| Message: "A section should have an associated repository" or "Could not load terminology" or "Section type not found in terminology"
+| Applies to: ``Section``
+| Course of action: Optional validation. Will report any section that does not specify a repository. Add a repository to the reported Section to resolve.
+
+Custom validations
+******************
+
+Users can write their own validation and register them either with the default validation or add it to their own validation class instance.
+
+A custom validation handler needs to ``yield`` a ``ValidationError``. See the ``validation.ValidationError`` class for details.
+
+Custom validation handlers can be registered to be applied on "odML" (the odml Document), "section" or "property".
+
+    >>> import odml
+    >>> import odml.validation as oval
+    >>>
+    >>> # Create an example document
+    >>> doc = odml.Document()
+    >>> sec_valid = odml.Section(name="Recording-20200505", parent=doc)
+    >>> sec_invalid = odml.Section(name="Movie-20200505", parent=doc)
+    >>> subsec = odml.Section(name="Sub-Movie-20200505", parent=sec_valid)
+    >>>
+    >>> # Define a validation handler that yields a ValidationError if a section name does not start with 'Recording-'
+    >>> def custom_validation_handler(obj):
+    >>>     validation_id = oval.IssueID.custom_validation
+    >>>     msg = "Section name does not start with 'Recording-'"
+    >>>     if not obj.name.startswith("Recording-"):
+    >>>         yield oval.ValidationError(obj, msg, oval.LABEL_ERROR, validation_id)
+    >>>
+    >>> # Create a custom, empty validation with an odML document 'doc'
+    >>> custom_validation = oval.Validation(doc, reset=True)
+    >>> # Register a custom validation handler that should be applied on all Sections of a Document
+    >>> custom_validation.register_custom_handler("section", custom_validation_handler)
+    >>> # Run the custom validation and return a report
+    >>> custom_validation.report()
+    >>> # Display the errors reported by the validation
+    >>> print(custom_validation.errors)
+
+Advanced Value features
+-----------------------
 
 Data type conversions
 *********************
@@ -1021,8 +1180,6 @@ converted to integer and then back to float::
 
 Links & Includes
 ****************
-Please note, that this section is outdated but still valid.
-
 Sections can be linked to other Sections, so that they include their defined
 attributes. A link can be within the document (``link`` property) or to an
 external one (``include`` property).
@@ -1054,8 +1211,6 @@ then set merge with the new object.
 
 Terminologies
 *************
-Please note, that this section is outdated but still valid.
-
 odML supports terminologies that are data structure templates for typical use cases.
 Sections can have a ``repository`` attribute. As repositories can be inherited,
 the current applicable one can be obtained using the

odml/property.py

@@ -573,7 +573,12 @@ def _values_cardinality_validation(self):
         Runs a validation to check whether the values cardinality
         is respected and prints a warning message otherwise.
         """
-        valid = validation.Validation(self)
+        # This check is run quite frequently so do not run all checks via the default validation
+        # but use a custom validation instead.
+        valid = validation.Validation(self, validate=False, reset=True)
+        valid.register_custom_handler("property", validation.property_values_cardinality)
+        valid.run_validation()
+
         val_id = validation.IssueID.property_values_cardinality
 
         # Make sure to display only warnings of the current property