Use dataclasses/pydantic instead of dicts?

[dhdaines]

This is obviously a (very) breaking change, but I find myself wishing that pdfplumber used something like Pydantic, dataclasses or even just plain old namedtuple for the various objects it returns (.chars, extract_words, etc, etc).

The main reason obviously is type checking - it will also help those of us in more recent British colonies who have an annoying habit of typing stroking_colour instead of stroking_color :) - but also the syntax is more concise and easier to read.

namedtuples have a certain efficiency advantage over dicts but I don't think this is the case for Pydantic or dataclasses.

Has this been considered?

[dhdaines]

EDIT: dataclass is as fast as namedtuple. Pydantic is slow. https://janhendrikewers.uk/pydantic_vs_protobuf_vs_namedtuple_vs_dataclasses.html

... but Pydantic V2 is considerably faster (would need to benchmark this, though).

[jsvine]

Thanks for raising this, @dhdaines. If there's a non-breaking, performance-stable way to do this, I'm all for it. Seems like it'd add some consistency, legibility, and rigor to an important part of the library.

@julian-r had a similar suggestion earlier this year, in #816

The idea with the "legacy getter layer" @julian-r mentioned, I think, is to prevent the breakage — so that people could still call, e.g., my_char["x0"] instead of having to call my_char.x0. This seems possible with dataclass, but possibly(?) not with namedtuple.

The main question then becomes performance.

[dhdaines]

I may be missing some obvious problem with this approach but it appears that a legacy getter is very easy to implement with nametuple:

from collections import namedtuple

PDFChar = namedtuple("PDFChar", ["x0", "x1", "top", "bottom", "text"])
PDFChar.__getitem__ = PDFChar.__getattribute__
pc = PDFChar(10, 20, 10, 20, "X")
print(pc.x0)
print(pc["x0"])

[jsvine]

I stand corrected, thanks! Some observations doing quick little demo, converting char objects to namedtuple and then running the test suite:

• It was relatively painless. clip_obj (used by crop) was the main method that needed rewriting, but it was fairly straightforward. A couple of the display methods, too, since they have conditionals using isinstance(obj, tuple). (I did, though, skip the test_convert.py tests, which felt like a distraction for now.)
• Adding new methods on namedtuple feels a bit clunky; maybe this is an argument for dataclass, particularly if we want to take advantage of subclassing.
• The changes I made led to a ~5% performance hit on the test suite ... but that's without any optimization at all. So I'm feeling somewhat optimistic on that front.

[dhdaines]

Nice! Yes, I suspect dataclass (with __slots__ to reduce memory usage) may be the best option. I think it is also more interoperable with Pydantic (and Pandas as you mention in #816).

[dhdaines]

except __slots__ doesn't really work well with dataclass until Python 3.10 :(

[jsvine]

Oy, good catch, thank you.