Retrieve large data that was split across multiple deals

[bajtos]

Quoting from https://filecoinproject.slack.com/archives/C02T827T9N0/p1718714837853519?thread_ts=1718710948.121159&cid=C02T827T9N0

So the way the pipeline works is as follows:

• Solana validator nodes writes out a large CAR file (~550GB) with a root payload cid.
• This CAR file is then split into smaller CAR files of 32GB size -> this is done without actually respecting the DAG structure, but just taking 1 CID at a time.
• Each of the smaller CAR files are sent to SPs to make deals.

So this means to capture the full dag we necessarily need to onboard all the CAR files to SPs that will serve retrievals, otherwise this breaks.

also

This is why it works:
https://github.com/filecoin-project/data-prep-tools/blob/main/docs/best-practices/car-first-then-split.md

Example CIDs from that thread:

• PieceCID baga6ea4seaqc7zmjswf4esktbdapqnzwanphet3g2ekppezzypodimfga6xiila
• Payload CIDs:
  • bafyreih4wq2ljuzhnn6pzl7tny7khzekqjx7yp6h5rvfbx2hrwrtp6mpcq
  • bafyreiedsc7cjq7ypu4fwmnrrttdquewlpgnpirjaubf5rrqexl4fnbeem
  • bafyreib3ik6guatvlcpy2oq7fgiavs6nbsc4i7ormq3gex4www53sdb364
  • bafyreihp4ocqnpbdeywmcpbukeutbrj3x5v2truzh4fkmyelninv6abfoa
  • bafyreihjk6o5jelb2nyvaltgor6rql5fac7lfucxnczegjibxzlubsfj5i
  • bafyreifazskco4pkbopnxeouanepbq26suth2gml6ufepcx4tcqq3j4deq