Cleanups for DSN Data Retrieval

[teor2345]

This is a list of remaining low-priority protocol team tasks for implementing object mapping and retrieval from the DSN.

Cleanups

• Create a separate gateway info RPC on the node server and gateway client
• Rewrite object_fetcher.rs to use regular positions, not source positions (optional - we improved the source position code instead)
• Only return mappings from find_last_archived_block() if we’re starting mapping at that exact block

Potential Object Mapping Changes

• add acknowledgements to provide backpressure and make sure the client gets all the mappings (at least from the point where they connect) - awaiting feedback on whether this is needed

mapping generation completed

• Extract mappings to objects in system.remark, system.remark_with_event, and seed_history extrinsics
• Implement incremental mapping generation (piece data is still only generated once the segment is full) - Generate object mappings incrementally #3100
  • Update archiver tests
• Expose mappings as an RPC subscription
  • Add a filtered subscription by object hash
  • Check if these RPC subscriptions are safe before allowing them - Deny unsafe object mapping subscriptions #3118
• Add a CLI option to generate all mappings from the start of the chain (archiving is currently skipped during initial sync) - Add a --create-object-mappings node CLI option #3125
• Add a new command line argument (or an optional height) where the node starts creating mappings. This will speed up recovery after downtime when there are lots of archived segments/mappings.
  • Add a block number to object mappings #3204
  • Create mappings from a specified block number #3194

The mapping indexer/storage is a separate component, which is out of scope for the protocol team.

Object Fetching

object fetching completed

• subspace-networking piece request-response protocols #3099
• Get pieces in batches #3126
• Implement a piece getter using PieceProvider
  • Change it to take a list of piece indexes
• Create a separate binary which takes a mapping via RPC, and fetches that object
  • Create a Rust crate which implements an abstract object fetching API, using a generic piece fetcher
  • Binary without any RPC methods or configuration - Initial subspace-gateway binary which does nothing #3131
  • Add fetcher RPC - Add an initial gateway RPC crate #3139
  • Connect to DSN - Add DSN client to subspace-gateway binary #3137
    • Add listen on address for debugging
  • Pass the DSN connection to the PieceProvider, create an ObjectFetcher, and launch the RPC server - Implement gateway RPC launch and piece getting #3148
  • Connect to node to get network values
    • Use correct genesis hash
    • Do piece validation
  • Maximum object size on CLI
• Document this binary and its RPC
• Fetch multiple pieces during object reconstruction #3158
  • and during segment reconstruction: Fetch multiple pieces during object reconstruction #3158

Documentation

• Document the node RPCs and the extrinsic objects they expose
  • Document the create object mappings CLI option

Testing

low priority tests

Object Mappings

• Exit with an error if mappings can’t be created for the chosen block #3242
  • add tests for this

Object Fetching

• Fix offset calculation bugs in ObjectFetcher #3239
  • add tests for this

Preparation

• Add binary to Docker (requested by Carlos), based on bootstrap node dockerfile
  • Add gateway to snapshot builds in CI #3256
  • Upload gateway binary to artifacts and assets #3257
• Manual tests

Unit Tests

• Do we need tests for mapping extraction?
• Add node RPC tests
• Add gateway RPC tests
• Add object fetching tests