Model architecture: Share & story migration #3654

soyarsauce · 2019-08-20T05:49:50Z

As a map owner
I want my v8 map to load old v7 generated share links
So that I don't break old links

Other notes:

terriajs-server is already being used when loading share links, so it makes most sense to do this at the terriajs server.
dragging catalogs into browser
non-server-side-share-links (JSON in URL - this is unofficially supported, so do we officially support this? NO)
properties that v8 doesn't understand
catalog types that v8 doesn't understand
any viewstate-ey things that aren't implemented
any other config
(what other warnings we don't even need to throw warnings about?)
v7 shares with init sources shared

DoD:

Unit test written (if applicable)
Works on mobile (or deliberately disabled for mobile)
Code/peer reviewed (if didn't pair)
Relevant docs are updated
Previewable/deployed to dev (see separation of deployments)
Previewable/deployed to test (see separation of deployments)
(If a map-specific ticket) Issue linked to a release ticket

Acceptance Criteria:

Loading old share links (~0.0.05) into a v8 map works
Server passes some message to client so that the payload has a "this used to be v7"
A UI warning to show that we've detected a v7 link was loaded in and that it may not work as intended
v7 share data is applied at user strata level

As a second pass

v7 share links containing stories will turn into working v8 stories

Background context

We've versioned mobx branch shares to 8.0.0, while non-mobx is on 0.0.05 - we should use this information to transform and preserve functionality of old share links as much as possible

Wing Ho [2:41 PM]
so something that came up on gitter - is mobx going to break
`0.0.05` share links? i haven't looked at how `8.0.0` (mobx versioned constant)
differs but i imagine we do some transformation from 0.0.05->8.0.0 within
terriajs, before putting it through the mobx logic?

Stephen Davies [3:29 PM]
@soyarsauce I’m not sure we’ve really thought it out,
but I think the result will be no; all share links will be broken (edited) 


Wing Ho [3:31 PM]
yeah that's what i mean, we put the payload through some sort
of migration when we see 0.0.05 to transform it into mobx-eatable-format?

Stephen Davies [3:32 PM]
Possibly for buckets we control we could convert all
the share data to mobx-compatible while converting the catalog of an instance
But a lot of our share data is mixed Maybe IDs will save a lot of it and we can
preserve what items are enabled, the camera angle and the time

Wing Ho [3:34 PM]
i wouldn't want to change the content in buckets themselves -
they should remain immutable, and we bundle the transformer for 0.0.05->8.0.0
either in terriajs-server or in terriajs itself

Stephen Davies [3:38 PM]
Yeash, I guess a simple transformer that would work is
iterate through all the items, guess ids for shared items without ids, apply a
transformation specific to the item’s type that tries to preserve as much of the
state about the catalog item while outputting new share attributes

Wing Ho [3:44 PM]
:+1:  sounds like a post-DT-pre-bigbangmerge piece of work

The text was updated successfully, but these errors were encountered:

soyarsauce · 2020-08-17T05:36:44Z

Also needs to work for story

nf-s · 2020-09-09T11:50:23Z

Update!

I have basic share-conversion working with enabled csv catalog items + common properties which haven't changes (camera position...)

PRs

TerriaJS - Share conversion messages #4735
TerriaJS-server - Setup Typescript! (+v7-v8 share link conversion) terriajs-server#119
Catalog-converter - Add options for share json support catalog-converter#11

nf-s · 2020-09-11T06:10:10Z

Update 2!

It is a bit ugly, but a start!

nf-s · 2020-09-11T06:34:38Z

TODO:

Test more examples of catalogs (test stories)
Write unit tests
Support client-side?
Handle URL initSources?

nf-s · 2020-09-13T10:39:23Z

Added story support!

nf-s · 2020-09-14T04:48:05Z

Order of things todo:

Fix test errors in TerriaJS PR ...
Terriajs PR - Share conversion messages #4735
Write share data tests in catalog-converter
Catalog-converter PR - Add options for share json support catalog-converter#11
Fix terriajs-server test suite - Setup Typescript! (+v7-v8 share link conversion) terriajs-server#119
TerriaJS-server PR - Setup Typescript! (+v7-v8 share link conversion) terriajs-server#119
Review Multi-tenancy PR - Multi tenant terriajs-server#120

Related issues

Post-static page work catalog-converter#7

nf-s · 2020-09-17T06:36:59Z

For my assumptions on share/catalog structure - see TerriaJS/catalog-converter#11 (comment)

nf-s · 2020-09-17T13:08:10Z

Update!

Works with user added data (in or not in workbench) - but it doesn't work with nested user added groups (eg WMS-group).
To fix this I need to go through and convert IDs

soyarsauce · 2020-09-23T04:17:05Z

To assist with the ID problem perhaps we need to think about allowing for "previous known IDs or names" when we are migrating catalogs - just a fleeting thought

soyarsauce · 2020-09-24T01:34:37Z

See: #4774

na9da · 2020-09-24T04:16:39Z

Missing properties for Pacific Map (spc.json)

      # These ones are deprecated and can be written instead using `filterQuery`. 
      # Probably not worth converting as the item gets used only in a few places (Rowan)
      1 allowEntireWmsServers (CKAN)
      1 filterByWmsGetCapabilities (CKAN)
      1 includeCsv (CKAN, deprecated)
      1 includeEsriFeatureServer (CKAN, deprecated)
      1 includeEsriMapServer (CKAN, deprecated)
      1 includeGeoJson (CKAN, deprecated)
      1 includeKml (CKAN, deprecated)
      1 includeWfs (CKAN, deprecated)
      1 includeWms (CKAN, deprecated)


      7 isCsvForCharting (CSV, not ported)
      1 minimumMaxScaleDenominator (CKAN, not ported)

Missing properties for DE Africa (africa.json)

      5 featureInfoTemplate.formats
     17 ignoreUnknownTileErrors
     12 preserveOrder
     17 tileErrorThresholdBeforeDisabling

DE Aus (dea-maps.json)

      3 chartDisclaimer (add support in catalog-converter)
     18 featureInfoTemplate.formats
      3 id
     58 ignoreUnknownTileErrors
     53 leafletUpdateInterval (catalog-converter#9)
      3 legendUrl
      2 legendUrls
     29 preserveOrder
      3 shortReport
      1 template
      1 zoomOnEnable

DE Aus (terria-cube.json)

      1 custodian
      2 tryToLoadObservationData

na9da · 2020-09-24T04:17:34Z

ignoreUnknownTileErrors & tileErrorThresholdBeforeDisabling will be fixed by - #4730

nf-s · 2020-09-24T04:28:11Z

V7 catalog item properties missing from `catalog-converter`

(or missing entirely from V8)

Looks good!

We can ignore Pacific Map CSV errors - those layers are replaced by SDMX

Catalog Member

id ?
shortReport

WMS

legendUrl
zoomOnEnable
featureInfoTemplate.formats
leafletUpdateInterval (Fixed Add leafletUpdateInterval trait for wms items. catalog-converter#9)
ignoreUnknownTileErrors (Handle tile errors #4730)
chartDisclaimer (add support in catalog converter)
legendUrls
tileErrorThresholdBeforeDisabling (Handle tile errors #4730)

Group

custodian
preserveOrder

SOS

tryToLoadObservationData (not ported)

CKAN

includeCsv
includeGeoJson
includeKml
includeWms
includeWfs
includeEsriMapServer
includeEsriFeatureServer
filterByWmsGetCapabilities
minimumMaxScaleDenominator
allowEntireWmsServers

nf-s · 2020-09-24T05:58:07Z

@na9da :

So except for the CKAN & pacific map CSV and a few that I have marked - all the others need porting.

soyarsauce · 2020-09-24T06:43:14Z

id ?

Will likely be incorporated in some way via #4774

nf-s · 2020-09-24T10:57:30Z

I have written a big list of properties needed for WMS conversion - TerriaJS/catalog-converter#12

nf-s · 2020-09-25T08:00:30Z

Other catalog types needed for pacific map (through CKAN)

CSV
GeoJSON

CKAN issues

Nested groups
WMS groups...

nf-s · 2020-09-29T10:14:22Z

New notification

nf-s · 2020-10-12T02:33:16Z

Change "Warning" to "Notice"
Change "Warnings" to "Technical details"

See Share migration - warning text update issue

soyarsauce · 2020-10-20T03:04:08Z

(for @steve9164)
3 ways for share fixes in saas

v7-v8 catalog conversion. as long as you never want to change. no catalog editor.
fix share keys, with alias IDs.
actually fixed being able to move things. "full instant v8 catalog". or the magda-terria-api

nf-s · 2020-10-22T04:31:36Z

Note initializationUrls aren't dealt with at the moment. They are passed from v7-v8 shares, but v8 terriajs can't do anything with a v7 initializationUrl

nf-s · 2020-10-22T04:39:27Z

I'm going to close this, and turn all remaining tasks into separate issues.
This is closed by TerriaJS/catalog-converter#11

nf-s · 2020-10-22T04:43:33Z

Id issue temporarily solved by #4887

soyarsauce added the Version 8 formerly MobX label Aug 20, 2019

soyarsauce added this to the Prioritized New Model Architecture Issues milestone Aug 20, 2019

tephenavies mentioned this issue Apr 6, 2020

Local storage migration #4206

Open

rowanwins assigned nf-s Aug 17, 2020

soyarsauce changed the title ~~Model architecture: Share migration~~ Model architecture: Share & story migration Aug 17, 2020

soyarsauce added the High priority label Aug 27, 2020

This was referenced Sep 9, 2020

Setup Typescript! (+v7-v8 share link conversion) TerriaJS/terriajs-server#119

Merged

Add options for share json support TerriaJS/catalog-converter#11

Merged

Support share JSON TerriaJS/catalog-converter#2

Closed

Share conversion messages #4735

Merged

soyarsauce added the Epic label Sep 14, 2020

soyarsauce mentioned this issue Sep 24, 2020

Model architecture: "Alias IDs" or shareKeys #4774

Closed

nf-s closed this as completed Oct 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Model architecture: Share & story migration #3654

Model architecture: Share & story migration #3654

soyarsauce commented Aug 20, 2019 •

edited by nf-s

Loading

soyarsauce commented Aug 17, 2020

nf-s commented Sep 9, 2020 •

edited

Loading

nf-s commented Sep 11, 2020 •

edited

Loading

nf-s commented Sep 11, 2020 •

edited

Loading

nf-s commented Sep 13, 2020

nf-s commented Sep 14, 2020 •

edited

Loading

nf-s commented Sep 17, 2020 •

edited

Loading

nf-s commented Sep 17, 2020 •

edited

Loading

soyarsauce commented Sep 23, 2020

soyarsauce commented Sep 24, 2020

na9da commented Sep 24, 2020 •

edited

Loading

na9da commented Sep 24, 2020

nf-s commented Sep 24, 2020 •

edited

Loading

nf-s commented Sep 24, 2020

soyarsauce commented Sep 24, 2020 •

edited

Loading

nf-s commented Sep 24, 2020 •

edited

Loading

nf-s commented Sep 25, 2020

nf-s commented Sep 29, 2020

nf-s commented Oct 12, 2020 •

edited

Loading

soyarsauce commented Oct 20, 2020

nf-s commented Oct 22, 2020

nf-s commented Oct 22, 2020

nf-s commented Oct 22, 2020

Model architecture: Share & story migration #3654

Model architecture: Share & story migration #3654

Comments

soyarsauce commented Aug 20, 2019 • edited by nf-s Loading

DoD:

Acceptance Criteria:

Background context

soyarsauce commented Aug 17, 2020

nf-s commented Sep 9, 2020 • edited Loading

Update!

PRs

nf-s commented Sep 11, 2020 • edited Loading

Update 2!

nf-s commented Sep 11, 2020 • edited Loading

nf-s commented Sep 13, 2020

nf-s commented Sep 14, 2020 • edited Loading

nf-s commented Sep 17, 2020 • edited Loading

nf-s commented Sep 17, 2020 • edited Loading

Update!

soyarsauce commented Sep 23, 2020

soyarsauce commented Sep 24, 2020

na9da commented Sep 24, 2020 • edited Loading

na9da commented Sep 24, 2020

nf-s commented Sep 24, 2020 • edited Loading

V7 catalog item properties missing from catalog-converter

Catalog Member

WMS

Group

SOS

CKAN

nf-s commented Sep 24, 2020

soyarsauce commented Sep 24, 2020 • edited Loading

nf-s commented Sep 24, 2020 • edited Loading

nf-s commented Sep 25, 2020

Other catalog types needed for pacific map (through CKAN)

CKAN issues

nf-s commented Sep 29, 2020

nf-s commented Oct 12, 2020 • edited Loading

soyarsauce commented Oct 20, 2020

nf-s commented Oct 22, 2020

nf-s commented Oct 22, 2020

nf-s commented Oct 22, 2020

soyarsauce commented Aug 20, 2019 •

edited by nf-s

Loading

nf-s commented Sep 9, 2020 •

edited

Loading

nf-s commented Sep 11, 2020 •

edited

Loading

nf-s commented Sep 11, 2020 •

edited

Loading

nf-s commented Sep 14, 2020 •

edited

Loading

nf-s commented Sep 17, 2020 •

edited

Loading

nf-s commented Sep 17, 2020 •

edited

Loading

na9da commented Sep 24, 2020 •

edited

Loading

nf-s commented Sep 24, 2020 •

edited

Loading

V7 catalog item properties missing from `catalog-converter`

soyarsauce commented Sep 24, 2020 •

edited

Loading

nf-s commented Sep 24, 2020 •

edited

Loading

nf-s commented Oct 12, 2020 •

edited

Loading