Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(xlsx): XLSX emits std minified .text_as_html #3558

Merged
merged 6 commits into from
Oct 17, 2024
Merged

Conversation

scanny
Copy link
Collaborator

@scanny scanny commented Aug 22, 2024

Summary
Eliminate historical "idiosyncracies" of table.metadata.text_as_html HTML introduced by partition_xlsx(). Produce minified .text_as_html consistent with that formed by chunking.

Additional Context

  • XLSX .text_as_html is minified (no extra whitespace or thead, tbody, tfoot elements).
  • table.text is clean-concatenated-text (CCT) of table.

@scanny scanny requested a review from Coniferish August 22, 2024 18:40
]


def test_partition_xlsx_with_find_subtables_False_and_infer_table_structure_False_works():
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a test case for find_subtable=True, infer_table_structure=False?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd actually be inclined not to have any tests like this :) This test boils down to "it doesn't raise for this arg combination", which is an impoverished test at best.

But I found a bug while I was in there on the html-minification and needed a test that triggered that bug so I could confirm my fix and defend against regression.

I think the broader remedy would be to break up partition_xlsx() a bit, like into a _XlsxPartitioner that we could write isolated unit tests for, but I didn't want to get into the extra scope.

Anyway, as it turns out, that particular case is handled by test_partition_xlsx_infer_table_structure() on line 95.

Copy link
Collaborator

@Coniferish Coniferish Aug 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me then. I had to look up what these parameters both did and wasn't positive on their interaction. It did cross my mind to add find_subtable to the docstring, but I assumed you would be against that. The docs might need to be updated to reflect that xlsx uses infer_table_structure, though, because I did query the ai search on the site and it said "The infer_table_structure parameter is not directly applicable to XLSX files in Unstructured."

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think both those have a place in the docstring. Definitely find_subtable, since it's XLSX-specific. infer_table_structure too because it's used in the partitioner code.

I've added find_subtable. infer_table_structure was already there. Sounds like the AI got that wrong.

Copy link
Collaborator

@Coniferish Coniferish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@scanny scanny enabled auto-merge August 22, 2024 19:36
@scanny scanny added this pull request to the merge queue Oct 17, 2024
Merged via the queue into main with commit c85f29e Oct 17, 2024
41 checks passed
@scanny scanny deleted the scanny/xlsx-minify-html branch October 17, 2024 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants