Use temporary SQLite db instead of list[dict]

[mikegerber]

Aug 02 07:00:54 b-pc30533 kernel: Out of memory: Killed process 4030869 (mods4pandas) total-vm:70463740kB, anon-rss:28581044kB, file-rss:1232kB, shmem-rss:4kB, UID:1000 pgtables:136832kB oom_score_adj:0

That's a whopping 28 GB memory after reading just 22% of the data...

→ Need a more memory-efficient way to handle this.

---

Progress:

• page_info
• Deal with existing files
• ValueError: No structMap[@type='PHYSICAL'] found (but not a multivolume work) (example on lx0246
• column names case insensitive
• Fix tests
• Convert to Parquet again
• Compare old files with new ones
  • Need to use JSON types for sets etc.
  • Need tests for these, it's a recurring issue
• alto_info

[mikegerber]

Considering all options, looks like iteratively build a SQLite database would be the best option (or at least worth trying)

[mikegerber]

I've done some experiments, and the temporary SQLite db seems to be the way to go.

[mikegerber]

Writing the records out to SQLite works nicely, barely any memory consumption.

Not terribly happy with the code and it now needs work to convert to Parquet, but overall this seems to be an elegant and sufficiently efficient solution.

[mikegerber]

Tests are still fine in this state as they don't read the files.

[mikegerber]

SQLite DBs are now converted to Parquet after creation.

[mikegerber]

In the previous version we had array columns:

>>> mods_info_df["classification-ZVDD"].iloc[0]
array(['Tibetica', 'Ostasiatica', 'Historische Drucke'], dtype=object)

:-\

[mikegerber]

alto_info/alto4pandas is done now, too.