Exposing geometries to Lua

[cldellow]

tl;dr: I want to use tilemaker to transform a PBF to a workable set of GeoJSON files for processing in another programming language. Expand for the full use case, if interested.

I am building yet-another-trails-app on OSM data.

I started using tilemaker because I wanted vector tiles. Since then, I've begun to treat tilemaker as a general-purpose runtime for processing spatial data: it provides parallelism, a scriptable interface, and can intelligently load only those parts of the PBF that I specify via node_keys and way_keys. It's a pretty satisfying developer experience!

My first such use was to produce a SQLite database, usable for autosuggest. I have a little file_append.lua module that lets me safely append to a file from multiple threads. I coupled that with rxi's json.lua module and Wikidata's QRank database to produce a JSON file that then gets indexed by SQLite. It worked pretty well, I think.

Now I want to go further, and spit out GeoJSON of certain elements contained in parks: route=hiking relations, highway=* ways, natural=peak nodes, water=lake areas, amenity=parking areas. I'd then post-process these, ultimately ending up with a canonicalized set of trails: each trail would have a start point, an end point, be normalized into either a loop, out-and-back or point-to-point route. In many cases, trails aren't described by route=hiking relations, so I'd also infer trails using a set of heuristics.

I think others might find this useful, too, for example, #556

I've explored a couple options, and am curious if one of them might be reasonable to include in tilemaker:

1. Don't use tilemaker - use Overpass Turbo, osmium, or OSMExpress, or similar.

I didn't explore this too much. I think the use case is narrow enough (export a set of OSM entities), that there's not too much benefit in any one tool over the other, so I defaulted to wanting to use fewer tools. I also liked the idea of re-using tilemaker as I have reusable predicates for filtering (e.g. "is this element contained within a national park?") and it'd mean we use the same geometry correction algorithms, so elements will line up exactly if they are later rendered on the same map.

2. Make --output foo.geojson dump the output objects (or add a --dump flag that did similar)

Discussed briefly in #630 (comment). This felt like a bit of a hack: output is tile-focused, has zoom levels, simplification, etc, this code path was ignoring all of that.

It also means that output objects have to be buffered in memory until the end of tilemaker's run. But for the export case, streaming them as you see them, and not retaining them is fine.

3. Add a GeoJSON(bool area) function to osm_lua_processing

The idea here is that we just need a serialization format to pass the data to Lua. GeoJSON seems to fit the bill.

I'm imagining this function would only return a geometry, not a feature. That is, it'd return:

{
  "type": "Point",
  "coordinates": [125.6, 10.1]
}

not

{
  "type": "Feature",
  "geometry": {
    "type": "Point",
    "coordinates": [125.6, 10.1]
  },
  "properties": {
    "name": "Dinagat Islands"
  }
}

The Lua code could either use just the geometry, or wrap it in a feature with properties, if that was appropriate for their need.

I've done some more prototyping with (3) and it's feeling pretty good.

Does this feel like it'd be a good fit for mainline tilemaker? If yes, there are some implementation details I'd want to run past you, but I figured I'd feel you out on the broad strokes question first.

[cldellow]

This might also get us closer to allowing a user to do #514 entirely in Lua.

With this change, the user would be able to produce a GeoJSON for use on a future run. If we went further and exposed a LayerAsGeoJSON(...) function, then the user could dynamically add nodes/ways/areas based on their own Lua computations. That would also address #252

From the point of view of the user, that might be pretty attractive. From the point of view of a tilemaker developer, this would be a bit weird, because now we're adding synthetic things to the OSMStore. I think it'd be technically fine, since we'd just treat them as materialized geometries, but it might raise the question of what should their ID be, if include-ids was set to true? Maybe such a function would just update the GeoJSON store, I guess.