Tools Overview

At cividi we are combining tools from the Open Web, Geospatial, Frictionless Data communities and ecosystems.

To do so we use the following tools and approaches for producing meaningful maps and data driven analysis.

Local/Development Stage

• Large Datasets: SQL-Query on a local SpatiaLite Database File, e.g. for BfS Datasets
• ETL Library:
  • Tool: DataFlows Python Library by Datopian
  • Tutorial: https://github.com/datahq/dataflows/blob/master/TUTORIAL.md
  • Builtin Processors: https://github.com/datahq/dataflows/blob/master/PROCESSORS.md
  • Library, the setzkasten of cividi processors
• Local ETL Runner + declarative flows with YAML input:
  • Tool: Datapackage Pipelines
  • Documentation/Tutorial: https://github.com/frictionlessdata/datapackage-pipelines
    • List pipelines: dpp
    • Run a pipeline: dpp run ./pipeline-name
• Validation:
  • done automatically by DataFlows, for manually checks:
  • frictionless-py with frictionless validate
  • data-cli with data validate, also interacts with DataHub.io API
• Styling:
  • Key Mappings for supported Geometry:
    • Polygons and LineStrings: GeoJSON with Simple Styles
    • Circles: Leaflet Circles
  • Legend entries (datapackage.views.spec.legend): SVG based
• Output: Snapshots compatible with Gemeindescan
  • Currently via a custom Snapshot Exporter
  • hopefully soon with official GeoJSON support in DataFlows and adjacent libraries

Production

Currently under evaluation: Pipeline Runner