chore: add ingest diagrams and explanation of flow to ingest README (#…

…1158)

CHANGELOG.md

@@ -1,4 +1,4 @@
-## 0.10.5-dev0
+## 0.10.5-dev1
 
 ### Enhancements
 * Create new CI Pipelines
@@ -8,6 +8,7 @@
 ## 0.10.3
 * Adds ability to reuse connections per process in unstructured-ingest
 * Pass ocr_mode in partition_pdf and set the default back to individual pages for now
+* Add diagrams and descriptions for ingest design in the ingest README
 
 ### Features
 

unstructured/__version__.py

@@ -1 +1 @@
-__version__ = "0.10.5-dev0"  # pragma: no cover
+__version__ = "0.10.5-dev1"  # pragma: no cover

unstructured/ingest/README.md

@@ -2,8 +2,7 @@
 
 ## The unstructured-ingest CLI
 
-The unstructured library includes a CLI to batch ingest documents from (soon to be
-various) sources, storing structured outputs locally on the filesystem.
+The unstructured library includes a CLI to batch ingest documents from various sources, storing structured outputs locally on the filesystem.
 
 For example, the following command processes all the documents in S3 in the
 `utic-dev-tech-fixtures` bucket with a prefix of `small-pdf-set/`.
@@ -79,3 +78,23 @@ In checklist form, the above steps are summarized as:
   - [ ] Else if `.preserve_download` is `False`, documents downloaded to `.download_dir` are removed after they are **successfully** processed during the invocation of `MyIngestDoc.cleanup_file()` in [process_document](unstructured/ingest/doc_processor/generalized.py)
   - [ ] Does not re-download documents to `.download_dir` if `.re_download` is False, enforced in `MyIngestDoc.get_file()`
   - [ ] Prints more details if `--verbose` in ingest CLI, similar to [unstructured/ingest/connector/github.py](unstructured/ingest/connector/github.py) logging messages.
+
+## Design References
+
+`unstructured/ingest/main.py` is the entrypoint for the `unstructured-ingest` cli. It calls the cli Command as fetched from `cli.py` `get_cmd()`.
+
+`get_cmd()` aggregates all subcommands (one per connector) as defined in the cli.cmd module. Each of these per-connector commands define the connector specific options and import the relevant common options. They call out to the corollary cli.runner.[CONNECTOR].py module.
+
+The runner is a vanilla (not Click wrapped) Python function which also explicitly exposes the connector specific arguments. It instantiates the connector with aggregated options / configs and passes it in call to `process_documents()` in `processor.py`.
+
+![unstructured ingest cli diagram](img/unstructured_ingest_cli_diagram.jpg)
+
+Given an instance of BaseConnector with a reference to its ConnectorConfig (BaseConnectorConfig and StandardConnectorConfig) and set of processing parameters, `process_documents()` instantiates the Processor class and calls its `run()` method.
+
+The Processor class (operating in the Main Process) calls to the connector to fetch `get_ingest_docs()`, a list of lazy download IngestDocs, each a skinny serializable object with connection config. These IngestDocs are filtered (where output results already exist locally) and passed to a multiprocessing Pool. Each subprocess in the pool then operates as a Worker Process. The Worker Process first initializes a logger, since it is operating in its own spawn of the Python interpreter. It then calls the `process_document()` function in `doc_processor.generalized.py`. 
+
+The `process_document()` function is given an IngestDoc, which has a reference to the respective ConnectorConfigs. Also defined is a global session_handler (of type BaseSessionHandler). This contains any session/connection relevant data for the IngestDoc that can be re-used when processing sibling IngestDocs from the same BaseConnector / config. If the value for the session_handle isn't assigned, a session_handle is created from the IngestDoc and assigned to the global variable, otherwise the existing global variable value ("session") is leveraged to process the IngestDoc. The function proceeds to call the IngestDoc's `get_file()`, `process_file()`, `write_result()`, and `clean_up()` methods.
+
+Once all multiprocessing subprocesses complete a final call to the BaseConnector `clean_up()` method is made.
+
+![unstructured ingest processing diagram](img/unstructured_ingest_processing_diagram.jpg)