Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow site developers to customize the ATLAS ingestion pipeline #8

Open
jacobwegner opened this issue Aug 14, 2020 · 0 comments
Open

Comments

@jacobwegner
Copy link
Contributor

jacobwegner commented Aug 14, 2020

As a follow on to #7, prepare_db assumes that all possible annotations are ingested, and that we're always starting from 0.

We likely want to provide something at the site level so that developers can customize the pipeline (via a hookset or something like the python social auth pipelines functionality).

This issue and #7 may also be useful in further decoupling the "ingestion" process from the deployed ATLAS. E.g., we run scaife-viewer/scaife-search-indexer on a really powerful VM so we can parallelize ingestion of ElasticSearch documents.

There may be a use case where developers / curators work together to "bake" an ATLAS data set (whether into a Docker image or as a tarball, etc) that could be packaged and deployed with a backend.

We may also want to decouple "extraction" from "ingestion"; e.g., we extract data sets as standoff annotation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant