StackExchange Watcher does not correctly grabs new dumps

[benoit74]

Due to changes in StackExchange processes, our watcher does not grab new dumps properly.

Originally, all dumps where pushed to https://archive.org/details/stackexchange

Latest dumps have dedicated URLs now:

• https://archive.org/details/stackexchange_20240630
• https://archive.org/details/stackexchange_20240630_revised
• https://archive.org/details/stackexchange_20240930

Looks like we can search for these new identifiers with this URL: https://archive.org/services/search/beta/page_production/?user_query=subject:%22Stack%20Exchange%20Data%20Dump%22%20creator:%22Stack%20Exchange,%20Inc.%22&hits_per_page=1&page=1&sort=date:desc&aggregations=false&client_url=https://archive.org/search?query=subject%3A%22Stack+Exchange+Data+Dump%22+creator%3A%22Stack+Exchange%2C+Inc.%22

[benoit74]

This is blocked by the fact that Sites.xml is not provided anymore in the datadumps.

I've opened an issue upstream: https://meta.stackexchange.com/questions/404002/sites-xml-is-not-present-anymore-in-stackexchange-data-dumps

[benoit74]

I've pushed current changes (not complete at all, but already capable to retrieve the most recent dump) to https://github.com/openzim/zimfarm/tree/new_dumps_url ; waiting for SE answer on upstream issue

[benoit74]

It seems now clear that Sites.xml will not be provided anymore, or at least it takes too long to have an official answer so let's move on and adapt. We need to adapt the watcher to not need Sites.xml anymore.

This is probably pretty easy for the wraper since we want to download/upload all archives, and in every dump there is already an XML file for that, see e.g. https://ia904506.us.archive.org/25/items/stackexchange_20240930/stackexchange_20240930_files.xml

Note as well that it looks like stackoverflow dump is not split into multiple files anymore.