Shared I/O library for scientific packages.

[pfebrer]

Hi! I'm a contributor to the sisl python package and I'm trying to understand what would be the openness of important scientific packages to have a common library dedicated to only input/output. Since pymatgen is one of the most used and starred packages (congratulations on the success! 😄), I would love it if you found the time to chime in on the discussion at https://gitlab.com/ase/ase/-/issues/1056#note_851538575 to give your POV about the idea.

Cheers and thanks in advance!

[mkhorton]

Hi @pfebrer, welcome!

I'll offer some quick thoughts here, in no particular order:

• I/O is very difficult to get right, as I'm sure you're aware. We have had a long-running issue/PR (see Abstract interface for Input classes #2211) about our own I/O interface.
  • Good I/O is fast, correct, compliant with any relevant standards (which are often very ill-specified in our field, or not correctly applied even if specifications exist), while also supporting whatever latest file formats are required (again, important in a scientific research context where things can move quickly).
• In pymatgen, we have the additional context in that this code is used to support high-throughput infrastructure for many people and this requires a certain stability of file format (specifically, the JSON representations of many core pymatgen classes, several of which provide de facto schemas for common calculation outputs).
• In general, I'm personally a fan of standardized libraries. I really like cclib for their efforts, I'm very optimistic about py4vasp as a first-party supported I/O library for the VASP code. However, the crucial consideration is what the long-term support and governance of such a project looks like.
  • Since I/O libraries are often at very foundation of other codes, it's important they are held at a high standard where possible, and are responsive to fixing issues, merging new features, and keeping code concise and easily installable.
  • Noting this, is not to say that we have solved these all these issues ourselves, but just that it's an important consideration.

With that said, I actually think the biggest barriers to a shared I/O library are not technical barriers, but are more related to project management barriers (e.g., how to collaborate across projects, how to ensure incentives are aligned appropriately, how to ensure contributors get appropriate recognition, how to ensure such a library is a net benefit and doesn't hold back research by introducing friction, how to resolve conflict, etc.)

On the pymatgen side, we already have much of the I/O we need, and support quite a variety of codes. This took many many years to even reach this coverage. Personally, I am quite in favour of a shared I/O library in theory (any reduction in the code that we have to maintain while still achieving the same research goals is a good thing!) but it does feel like a serious effort to undertake, and whether it is practically possible will very much depend on the details.

(Edited to add additional notes.)

[mkhorton]

Hi @ml-evs, I just wanted to let you know that optimade-python-tools had come up in discussion in the linked thread.

[ml-evs]

Thanks for looping me in @mkhorton, I will try to give an OPTIMADE perspective next week on that GitLab thread. We are also trying to think about this stuff from the infrastructure side as a potential MaRDA working group at the moment too (the idea being a registry of parsers/extractors with well-defined schemas for various data producing instruments/software).