pypdf moonshot extensions 🚀

[MartinThoma]

PDF is a crazy format with lots of different aspects. While PyPDF2 has a huge user base, the number of active contributors is still limited. That means that we need to prioritize which topics to work on.

In general, PyPDF2 is open to any contribution. But if a contribution gets merged it needs to be (1) finished enough to provide value to users (2) be clear enough to maintainers to take care of it - test help a lot!

People who want to support PyPDF2 will either take an issue they experience themselves, or they look in the open issues. That means feature requests which hang around for too long reduce the amount of contributions as they distract.

This discussion mainly links to issues which were closed due to too little support. If we have collected enough knowledge / enough people that can work on them, we can re-open them:

• Table Extraction: Closed after 6+ years without progress; camelot-py might be an alternative
• Digital signatures: Closed after 5+ years without progress; pyHanko might be an alternative
• Optional Content Groups: Closed after 8+ years without progress; to be honest I don't quite understand what those are needed for / in which scenarios those are relevant
• Fast Web View: Closed after 4+ years without progress
• Confining user selection to text blocks: Closed after 4+ years without progress
• Subscripts / superscripts text extraction

[MartinThoma]

One of the most important extensions would be proper PDF Form handling. I didn't close that issue as it's not a feature request, but a bug.

Adding FDF support would be a new feature. I have too little knowledge to even describe how this could be done well.
It was mentioned in #961

[pubpub-zz]

PDF Form Handling is implemented (data extraction and update) but signatures

[pubpub-zz]

FDF implementation is currently rejected

[JeffBarrette]

Concerning Optional Content Groups, I can speak to how it affects our workflows ...

PDFs have the ability to persist "layers" and "attributes". With "layers" you can toggle the visibility of elements in the PDF. If individual PDFs are appended using PYPDF2, this information gets stripped from those individual PDFs in the resulting, combined PDF.

I hope this helps,
Jeff

[ghost]

Similar to Jeff, I have interest in OCGs.

We currently us PyPDF2 to compile CAD-exported PDF files. These files contain OCGs, and it would be very benificial for us to write multiple output files with various OCGs turned off. For us, the ability to retain OCGs in the output file is less important than the ability to modify their visibility prior to writing the output file.

As a side note, I am extreamly happy with the work that has been done on PyPDF2 since April. It was good before, but now it is great!

Much appreciation,
Ethan

[pubpub-zz]

Proposed ideas about text extraction improvement:

• compute the physical box containing some text (must be on the same line)
• extract the text in the physical page order and with improved horizontal positioning/spacing
• add some filtering criteria: text within a physical box, text defined in a font and/or text size

[MartinThoma]

Does this solve the "compute the physical box containing some text" part?

[pubpub-zz]

Not from my point of view: they are complementary : the feature I propose will provide a way to identify the box. The propotype function I would propose would be:
page.get_box_containing(expr : Union[str, RegExp]) -> Rectangle :
the rectangle should be the 'exact/most accurate' box containing the text, expr will only be applied on the current line processed

the top/bottom could be then use as input in the #1252. Actually it is a first proposal for "text within a physical box". Actually the PR may be slightly adjusted add Font and Font_Size(scaled to page) in order to add some filtering capabilities

Also this rectangle width/height should be consistent with the characters width.

[pubpub-zz]

New proposed idea : image extraction

[MartinThoma]

Just in case you haven't seen it: https://github.com/py-pdf/PyPDF2/blob/main/PyPDF2/filters.py#L558

There are a couple of tests for it as well that show how to use it. What is missing is a proper integration.

Maybe a new read-only property for the PageObject:

@dataclass
class ImageFile:
    name: str
    contents: bytes

@property
def images(self) -> List[ImageFile]: ...

?

[pubpub-zz]

I did see it but from my point of view it is partial as the inline images are not extracted

[MartinThoma]

It's always interesting to me to see how other libraries do it:

• PyMuPDF: https://pymupdf.readthedocs.io/en/latest/recipes-images.html#how-to-extract-images-pdf-documents
• https://askubuntu.com/q/117143/10425
• https://stackoverflow.com/q/2693820/562769
• Django File class: https://docs.djangoproject.com/en/4.1/ref/files/file/ (no PDF relation, but might be interesting if we also create a file class)

[pubpub-zz]

Image extraction including inline images is implemented. Still some improvement required for some specific features/colorspaces and JBIG2 filter (#1989)

[pubpub-zz]

Performance issue about FloatObjects
#1371 (comment)

[pulse-mind]

I suggest to add incrementalSave
The goal is to add annotation on the PDF without breaking signatures...

[pubpub-zz]

From #232
many requests about Flatten Pdf Written should be added