detect document language across all partitioners

CHANGELOG.md

@@ -1,7 +1,8 @@
-## 0.10.20-dev9
+## 0.10.20
 
 ### Enhancements
 
+* **Add document level language detection functionality.** Adds the "auto" default for the languages param to all partitioners. The primary language present in the document is detected using the `langdetect` package. Additional param `detect_language_per_element` is also added for partitioners that return multiple elements. Defaults to `False`.
 * **Refactor OCR code** The OCR code for entire page is moved from unstructured-inference to unstructured. On top of continuing support for OCR language parameter, we also support two OCR processing modes, "entire_page" or "individual_blocks".
 * **Align to top left when shrinking bounding boxes for `xy-cut` sorting:** Update `shrink_bbox()` to keep top left rather than center.
 * **Add visualization script to annotate elements** This script is often used to analyze/visualize elements with coordinates (e.g. partition_pdf()).

example-docs/language-docs/eng_spa_mult.eml

@@ -0,0 +1,113 @@
+MIME-Version: 1.0
+Date: Wed, 4 Oct 2023 09:27:45 -0500
+Message-ID: <CABDvgF2Wpt9+eSO7zgMJZ2fQb=QZ__CS6N_Y+msnGwpKeg1a+A@mail.gmail.com>
+Subject: Test email with multiple languages
+From: John <[email protected]>
+To: John <[email protected]>
+Content-Type: multipart/alternative; boundary="000000000000a0666d0606e4cfb7"
+
+--000000000000a0666d0606e4cfb7
+Content-Type: text/plain; charset="UTF-8"
+Content-Transfer-Encoding: quoted-printable
+
+All human beings are born free and equal in dignity and rights. They are
+endowed with reason and conscience and should act towards one another in a
+spirit of brotherhood. All human beings are born free and equal in dignity
+and rights. They are endowed with reason and conscience and should act
+towards one another in a spirit of brotherhood. All human beings are born
+free and equal in dignity and rights. They are endowed with reason and
+conscience and should act towards one another in a spirit of brotherhood.
+All human beings are born free and equal in dignity and rights. They are
+endowed with reason and conscience and should act towards one another in a
+spirit of brotherhood. All human beings are born free and equal in dignity
+and rights. They are endowed with reason and conscience and should act
+towards one another in a spirit of brotherhood. All human beings are born
+free and equal in dignity and rights. They are endowed with reason and
+conscience and should act towards one another in a spirit of brotherhood.
+
+All human beings are born free and equal in dignity and rights. They are
+endowed with reason and conscience and should act towards one another in a
+spirit of brotherhood. "Todos los seres humanos nacen libres e iguales en
+dignidad y derechos y, dotados como est=C3=A1n de raz=C3=B3n y conciencia, =
+deben
+comportarse fraternalmente los unos con los otros. Todos los seres humanos
+nacen libres e iguales en dignidad y derechos y, dotados como est=C3=A1n de
+raz=C3=B3n y conciencia, deben comportarse fraternalmente los unos con los
+otros."
+
+All human beings are born free and equal in dignity and rights. They are
+endowed with reason and conscience and should act towards one another in a
+spirit of brotherhood. All human beings are born free and equal in dignity
+and rights. They are endowed with reason and conscience and should act
+towards one another in a spirit of brotherhood. All human beings are born
+free and equal in dignity and rights. They are endowed with reason and
+conscience and should act towards one another in a spirit of brotherhood.
+All human beings are born free and equal in dignity and rights. They are
+endowed with reason and conscience and should act towards one another in a
+spirit of brotherhood.
+
+All human beings are born free and equal in dignity and rights. They are
+endowed with reason and conscience and should act towards one another in a
+spirit of brotherhood. All human beings are born free and equal in dignity
+and rights. They are endowed with reason and conscience and should act
+towards one another in a spirit of brotherhood. All human beings are born
+free and equal in dignity and rights. They are endowed with reason and
+conscience and should act towards one another in a spirit of brotherhood.
+
+"Todos los seres humanos nacen libres e iguales en dignidad y derechos y,
+dotados como est=C3=A1n de raz=C3=B3n y conciencia, deben comportarse frate=
+rnalmente
+los unos con los otros. Todos los seres humanos nacen libres e iguales en
+dignidad y derechos y, dotados como est=C3=A1n de raz=C3=B3n y conciencia, =
+deben
+comportarse fraternalmente los unos con los otros."
+
+--000000000000a0666d0606e4cfb7
+Content-Type: text/html; charset="UTF-8"
+Content-Transfer-Encoding: quoted-printable
+
+<div dir=3D"ltr">All human beings are born free and equal in dignity and ri=
+ghts. They are endowed with reason and conscience and should act towards on=
+e another in a spirit of brotherhood. All human beings are born free and eq=
+ual in dignity and rights. They are endowed with reason and conscience and =
+should act towards one another in a spirit of brotherhood. All human beings=
+ are born free and equal in dignity and rights. They are endowed with reaso=
+n and conscience and should act towards one another in a spirit of brotherh=
+ood. All human beings are born free and equal in dignity and rights. They a=
+re endowed with reason and conscience and should act towards one another in=
+ a spirit of brotherhood. All human beings are born free and equal in digni=
+ty and rights. They are endowed with reason and conscience and should act t=
+owards one another in a spirit of brotherhood. All human beings are born fr=
+ee and equal in dignity and rights. They are endowed with reason and consci=
+ence and should act towards one another in a spirit of brotherhood.<br><br>=
+All human beings are born free and equal in dignity and rights. They are en=
+dowed with reason and conscience and should act towards one another in a sp=
+irit of brotherhood. &quot;Todos los seres humanos nacen libres e iguales e=
+n dignidad y derechos y, dotados como est=C3=A1n de raz=C3=B3n y conciencia=
+, deben comportarse fraternalmente los unos con los otros. Todos los seres =
+humanos nacen libres e iguales en dignidad y derechos y, dotados como est=
+=C3=A1n de raz=C3=B3n y conciencia, deben comportarse fraternalmente los un=
+os con los otros.&quot;<br><br>All human beings are born free and equal in =
+dignity and rights. They are endowed with reason and conscience and should =
+act towards one another in a spirit of brotherhood. All human beings are bo=
+rn free and equal in dignity and rights. They are endowed with reason and c=
+onscience and should act towards one another in a spirit of brotherhood. Al=
+l human beings are born free and equal in dignity and rights. They are endo=
+wed with reason and conscience and should act towards one another in a spir=
+it of brotherhood. All human beings are born free and equal in dignity and =
+rights. They are endowed with reason and conscience and should act towards =
+one another in a spirit of brotherhood.<br><br>All human beings are born fr=
+ee and equal in dignity and rights. They are endowed with reason and consci=
+ence and should act towards one another in a spirit of brotherhood. All hum=
+an beings are born free and equal in dignity and rights. They are endowed w=
+ith reason and conscience and should act towards one another in a spirit of=
+ brotherhood. All human beings are born free and equal in dignity and right=
+s. They are endowed with reason and conscience and should act towards one a=
+nother in a spirit of brotherhood.<br><br>&quot;Todos los seres humanos nac=
+en libres e iguales en dignidad y derechos y, dotados como est=C3=A1n de ra=
+z=C3=B3n y conciencia, deben comportarse fraternalmente los unos con los ot=
+ros. Todos los seres humanos nacen libres e iguales en dignidad y derechos =
+y, dotados como est=C3=A1n de raz=C3=B3n y conciencia, deben comportarse fr=
+aternalmente los unos con los otros.&quot;</div>
+
+--000000000000a0666d0606e4cfb7--

example-docs/language-docs/eng_spa_mult.html

@@ -0,0 +1,20 @@
+<!DOCTYPE html>
+<html>
+<body>
+    <p>
+        All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
+    </p>
+    <p>
+        All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. "Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros. Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros."
+    </p>
+    <p>
+        All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
+    </p>
+    <p>
+        All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
+    </p>
+    <p>
+        "Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros. Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros."
+    </p>
+</body>
+</html>

example-docs/language-docs/eng_spa_mult.md

@@ -0,0 +1,9 @@
+All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
+
+All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. "Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros. Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros."
+
+All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
+
+All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
+
+"Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros. Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros."

example-docs/language-docs/eng_spa_mult.org

@@ -0,0 +1,9 @@
+All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
+
+All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. "Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros. Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros."
+
+All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
+
+All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood. All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
+
+"Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros. Todos los seres humanos nacen libres e iguales en dignidad y derechos y, dotados como están de razón y conciencia, deben comportarse fraternalmente los unos con los otros."