Weekly Wiki Review

[bkarolina]

Individual discussion threads:

• Snapshots
• Tools Overview
• Pipeline Runner
• Automation of BFS based maps

How the review process works:

1. If you have a new or revised wiki page you want to get a review for

• Post a link to that page as a comment on this discussion thread
• Add it on the Home wiki page in the "Under Review" section

2. If you want to rate or comment on a posted wiki page

• Add a reaction to the original comment containing the wiki page link
• Or add a comment as a reply

General information on this repo and the overall process: see Wiki

[n0rdlicht]

Wiki/Snapshots

Status: Draft
Reason: Defining what a snapshot is, conceptually and technically

[bkarolina]

@n0rdlicht what do you think about including 'Intro' part what was used for hackathon? (https://hackmd.io/@n0rdlicht/Bk9gEqdYP)

[n0rdlicht]

Good idea, could be added as a "try it yourself" section.

Made one clarification that seems to come back every once in a while

• resources referenced in a view are a property of the view not view.spec

[n0rdlicht]

Wiki/Tools Overview

Status: Draft
Reason: Defining what tools are used for data analysis and automation

[bkarolina]

It needs bit of update regarding:

• ETL Library (frictionless framework)
• ETL Runner (frictionless framework)
• Validate (frictionless framework)
• Production - workflow and pipeline orchestartion orchestartion (prefect)

[n0rdlicht]

Fully agreed. Which part would you recommend to start with?

[bkarolina]

The old entries I would put into brackets/subsection as deprecated (to be clear, that we haven't migrate fully yet)./ Otherwise just put there new tools with links. This is just list of tools that we are using, not explaining how rather than where and why.

[bkarolina]

Wiki/Pipeline Runner

Status: Draft
Reason: Evaluating pipeline runner to use with a various notes, links.

[n0rdlicht]

👍 thanks for researching these and the good overview. Indeed I think Prefect matches what we need very well. And based on this evaluation add to the Tools Overview to outline how to move from local testing/development to production. @loleg any thoughts on Prefect as our automation tool? License is another one in the mix but same as Airflow.

Thoughts on production orchestration:

• Triggering Flows
  • Option Pull: A scheduled Prefect Flow fetching a List from Airtable/Trello/Google Sheets/... and updating it's status (queued, running, done, waiting for input, error, ...) back
  • Option Hybrid: Trigger the Pull Prefect Flow via the Prefect Server API
  • Option Push: Relying on the Prefect Server API to trigger Flows
• Task Library
  • Python Package for common Data and Prefect Flows to be loaded in Prefect Flows (cf. cividi Setzkasten)
  • test driven based on mock or open data, automated by CI
• Output Storage for Snapshot
  • S3?
• Preview: As part of authoring process a preview of the snapshot could be made available automatically before approval and publication

Technical Note: Currently can't get the server to run on M1 machines with the current Docker Dev Preview.

[bkarolina]

• Output Storage for Snapshot
• • Fedora Server (for testing phase, it could be as well at early production state, as in-between step before the snapshot is uploaded to sandbox)

[loleg]

I'm pretty sure I responded at some point over Slack, sorry to be redundant: I'm wary of automation which creates potential bottlenecks, such as overreliance on cloud hosted infrastructure, headache-inducing pricing structures, or inaccessibility to less technical users. I'd want to see a lot more of the team engaged on Prefect, seeing it put to more than just Snapshot-Factory uses, and definitely a clearer exploration of the licensing and self-hosting question before giving it my thumb.

[bkarolina]

Wiki/Automation of BFS based maps

Status: In progress
Reason: For now it is "library" of maps. This would be used for further discussion for automatiom, parametrization --> creating Setzkasten.

[Brieden]

Wiki/Map Engines Performance Test

Status: Finished
Reason: In order to keep an overview with an increasing number of new map engines and to classify our current approach.

[Brieden]

Wiki/Temporal Data Visualizacion

Status: Draft finished
Reason: Information + Draft to start the discussion about temporal or classified Geodata in Gemeindescan

[n0rdlicht]

Really like the idea of discrete filter sets.
Should we translate this into an issue on the Snapshot Specification? E.g. as started for Snapshot localisation.

[Brieden]

I agree that we can apply discrete filtering to a lot of use cases. For categories like (2011, 2012, ..., 2020) or (Leisure, Financial Services, ..., Industry) or even maybe (00:00, 01:00, 02:00, ... 24:00) should be enough space in the selection area.

[n0rdlicht]

Wiki/Localised Snapshots

Status: Proposal to Snapshot Specification
Reason: Adding localisation to snapshots for multi-language/region contexts

[Brieden]

Cookbook for BFS scaling on map

Status: Finished
Reason: Automated way to calc a function: Scaling radius from the high-dynamic-ranged BFS Data