bug/Unstructured unnecessarily "repairs" then falls back to OCR on extremely large documents #3815
Comments
|
If you must patch pdfminer.six, then please apply this fix, which fixes both the original problem and the problem that was introduced by patching it: pdfminer/pdfminer.six#1030 I can supply a PR. (note that it is not as simple as patching a single method, unfortunately, but should be doable) |
dhdaines
changed the title
|
+1 |
PhorstenkampFuzzy
added a commit
to PhorstenkampFuzzy/unstructured
that referenced
this issue
Dec 17, 2024
Removed redundent patch as it is already merged in pdfminer/pdfminer.six#885
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
TL;DR this is because Unstructured.io applies a "fix" to pdfminer.six which actually introduces another bug.
The fix (which I have tested here) is, until the bug is fixed in pdfminer.six, to not apply the patch, and to simply avoid the following versions of pdfminer.six which contain the bug: 20231228 and 20240706
The source of the problem is this change to pdfminer.six (which I think was submitted by unstructured?): pdfminer/pdfminer.six#885
This broke the parsing of long content streams, because due to the complex and not particularly useful buffering done by pdfminer.six's "parser" (actually a lexer), it is frequently the case that keywords are split across a buffer boundary. In my particular testcase (cannot share, sorry) this results in the "Detected invalid dictionary construct for PDFminer" error reported by Unstructured.io, because a dictionary ends up with an odd number of items as
falseis getting split intofalsande."Repairing" the PDF will not help this problem because there isn't anything wrong with the PDF, it just has some long content streams. So what happens is that (on my document of 1500+pages) unstructured falls back to OCR, which as you can imagine is ... suboptimal.
The text was updated successfully, but these errors were encountered: