Skip to content
This repository has been archived by the owner on Nov 7, 2019. It is now read-only.

Add Morph Run Custom Crawl Model #19

Open
b5 opened this issue Aug 24, 2017 · 0 comments
Open

Add Morph Run Custom Crawl Model #19

b5 opened this issue Aug 24, 2017 · 0 comments
Assignees

Comments

@b5
Copy link
Member

b5 commented Aug 24, 2017

From our meeting notes:

  1. we're allowed to make POST calls
    • we'll call some sort commit / finalize function
    • last but not least, POST overview data about the run to datatogether, including details on how >to download data.sqlite payload.
  2. Data Together accepts the post, stores the result, kicks off a task, which uses an "archiving library" (which currently doesn't exist) that conforms the the file, urls, & tabular data formats form the archiving library, downloads that data, posts it to IPFS, and then does a small dance. It also grabs the scraping script from.

Post-Run Payload

ident.archivers.co/jwt?access_token=foo&action=morph_run
api.archivers.co/morph

{
    "jwt": "eiwhofanerjnkgf.aweiofhaeioheaw.asnivocaneo",
    "morphRunId": "329awgknlvorapfevjiaowefha",
    "dateCompleted": "2017-03-9",
    "githubRepo" : "https://github.com/foo/bar",
    "originalUrl: "http://everything.gov.com.au.it.co.uk",
    "sqliteChecksum": "qafhq32ufh4a32uq4eraf3whaudrsfaiaewf"
}

cc @ebenp

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Development

No branches or pull requests

2 participants