Adds data source properties to git connectors

CHANGELOG.md

@@ -1,9 +1,9 @@
-## 0.10.17-dev16
+## 0.10.17-dev17
 
 ### Enhancements
 
 * **Improves `partition_xml` to be faster and more memory efficient when partitioning large XML files** The new behavior is to partition iteratively to prevent loading the entire XML tree into memory at once in most use cases.
-* **Adds data source properties to SharePoint, Outlook, Onedrive, Reddit, Slack, and DeltaTable connectors** These properties (date_created, date_modified, version, source_url, record_locator) are written to element metadata during ingest, mapping elements to information about the document source from which they derive. This functionality enables downstream applications to reveal source document applications, e.g. a link to a GDrive doc, Salesforce record, etc.
+* **Adds data source properties to SharePoint, Outlook, Onedrive, Reddit, Slack, DeltaTable, Github and Gitlab connectors** These properties (date_created, date_modified, version, source_url, record_locator) are written to element metadata during ingest, mapping elements to information about the document source from which they derive. This functionality enables downstream applications to reveal source document applications, e.g. a link to a GDrive doc, Salesforce record, etc.
 * **Add functionality to save embedded images in PDF's separately as images** This allows users to save embedded images in PDF's separately as images, given some directory path. The saved image path is written to the metadata for the Image element. Downstream applications may benefit by providing users with image links from relevant "hits."
 * **Azure Cognite Search destination connector** New Azure Cognitive Search destination connector added to ingest CLI.  Users may now use `unstructured-ingest` to write partitioned data from over 20 data sources (so far) to an Azure Cognitive Search index.
 * **Improves salesforce partitioning** Partitions Salesforce data as xlm instead of text for improved detail and flexibility. Partitions htmlbody instead of textbody for Salesforce emails. Importance: Allows all Salesforce fields to be ingested and gives Salesforce emails more detailed partitioning.

test_unstructured_ingest/expected-structured-output/github/LICENSE.txt.json

@@ -3,7 +3,15 @@
     "type": "Title",
     "element_id": "e3e5334b595ef9b648bf7f1f6c1a60c4",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/LICENSE.txt",
+        "version": "W/\"2c4f1ab8689a6dfef4ee7d13d4d935cb6663a7e4\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "LICENSE.txt"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/plain",
       "languages": [
         "eng"
@@ -15,7 +23,15 @@
     "type": "Title",
     "element_id": "8dc8800e5660b2558bb7f5f5416ca498",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/LICENSE.txt",
+        "version": "W/\"2c4f1ab8689a6dfef4ee7d13d4d935cb6663a7e4\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "LICENSE.txt"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/plain",
       "languages": [
         "eng"
@@ -27,7 +43,15 @@
     "type": "NarrativeText",
     "element_id": "fa3ff462f020dcadaf3c44b61f0df757",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/LICENSE.txt",
+        "version": "W/\"2c4f1ab8689a6dfef4ee7d13d4d935cb6663a7e4\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "LICENSE.txt"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/plain",
       "languages": [
         "eng"
@@ -39,7 +63,15 @@
     "type": "NarrativeText",
     "element_id": "70760316a66259dc346c891a2b964556",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/LICENSE.txt",
+        "version": "W/\"2c4f1ab8689a6dfef4ee7d13d4d935cb6663a7e4\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "LICENSE.txt"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/plain",
       "languages": [
         "eng"
@@ -51,7 +83,15 @@
     "type": "NarrativeText",
     "element_id": "1da9072633b5e4291608b205a664d5af",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/LICENSE.txt",
+        "version": "W/\"2c4f1ab8689a6dfef4ee7d13d4d935cb6663a7e4\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "LICENSE.txt"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/plain",
       "languages": [
         "eng"

test_unstructured_ingest/expected-structured-output/github/test.html.json

@@ -3,7 +3,15 @@
     "type": "Title",
     "element_id": "56a9f768a0968be676f9addd5ec3032e",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/test.html",
+        "version": "W/\"c63c8fc21d46d44de85a14a3ed4baec0348ce344\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "test.html"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/html",
       "page_number": 1
     },
@@ -13,7 +21,15 @@
     "type": "Title",
     "element_id": "d551bbfc9477547e4dce6264d8196c7b",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/test.html",
+        "version": "W/\"c63c8fc21d46d44de85a14a3ed4baec0348ce344\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "test.html"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/html",
       "page_number": 1,
       "link_urls": [
@@ -29,7 +45,15 @@
     "type": "Title",
     "element_id": "971b974235a86ca628dcc713d6e2e8d9",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/test.html",
+        "version": "W/\"c63c8fc21d46d44de85a14a3ed4baec0348ce344\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "test.html"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/html",
       "page_number": 1
     },
@@ -39,7 +63,15 @@
     "type": "Title",
     "element_id": "4112a488690bdbc1d39d5b78068eae9f",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/test.html",
+        "version": "W/\"c63c8fc21d46d44de85a14a3ed4baec0348ce344\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "test.html"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/html",
       "page_number": 1
     },
@@ -49,7 +81,15 @@
     "type": "NarrativeText",
     "element_id": "f89c9cf63bd2e72f560ee043d942a1e7",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/test.html",
+        "version": "W/\"c63c8fc21d46d44de85a14a3ed4baec0348ce344\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "test.html"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/html",
       "page_number": 1
     },
@@ -59,7 +99,15 @@
     "type": "NarrativeText",
     "element_id": "53a4db70c6d40ed5206711ed8a255e03",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/test.html",
+        "version": "W/\"c63c8fc21d46d44de85a14a3ed4baec0348ce344\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "test.html"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/html",
       "page_number": 1
     },
@@ -69,7 +117,15 @@
     "type": "Title",
     "element_id": "839973fba0c850f1729fad098b031203",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/test.html",
+        "version": "W/\"c63c8fc21d46d44de85a14a3ed4baec0348ce344\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "test.html"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/html",
       "page_number": 1
     },
@@ -79,7 +135,15 @@
     "type": "NarrativeText",
     "element_id": "b7db0dffb05f01f3f13d34420b82c261",
     "metadata": {
-      "data_source": {},
+      "data_source": {
+        "url": "https://raw.githubusercontent.com/dcneiner/Downloadify/master/test.html",
+        "version": "W/\"c63c8fc21d46d44de85a14a3ed4baec0348ce344\"",
+        "record_locator": {
+          "repo_path": "dcneiner/Downloadify",
+          "file_path": "test.html"
+        },
+        "date_modified": "2010-01-23T23:18:40"
+      },
       "filetype": "text/html",
       "page_number": 1
     },

unstructured/__version__.py

@@ -1 +1 @@
-__version__ = "0.10.17-dev16"  # pragma: no cover
+__version__ = "0.10.17-dev17"  # pragma: no cover

unstructured/ingest/connector/git.py

@@ -37,18 +37,35 @@ def filename(self):
     def _output_filename(self):
         return Path(self.partition_config.output_dir) / f"{self.path}.json"
 
+    @property
+    def record_locator(self) -> t.Dict[str, t.Any]:
+        record_locator = {
+            "repo_path": self.connector_config.repo_path,
+            "file_path": self.path,
+        }
+        if self.connector_config.branch is not None:
+            record_locator["branch"] = self.connector_config.branch
+        return record_locator
+
     def _create_full_tmp_dir_path(self):
         """includes directories in in the gitlab repository"""
         self.filename.parent.mkdir(parents=True, exist_ok=True)
 
+    def update_source_metadata(self, **kwargs):
+        raise NotImplementedError()
+
     @SourceConnectionError.wrap
     @BaseIngestDoc.skip_if_file_exists
     def get_file(self):
+        print(self)
         """Fetches the "remote" doc and stores it locally on the filesystem."""
         self._create_full_tmp_dir_path()
         logger.debug(f"Fetching {self} - PID: {os.getpid()}")
         self._fetch_and_write()
 
+    def _fetch_content(self) -> None:
+        raise NotImplementedError()
+
     def _fetch_and_write(self) -> None:
         raise NotImplementedError()
 

unstructured/ingest/connector/github.py

@@ -1,5 +1,6 @@
 import typing as t
 from dataclasses import dataclass
+from datetime import datetime
 from urllib.parse import urlparse
 
 import requests
@@ -10,6 +11,7 @@
     SimpleGitConfig,
 )
 from unstructured.ingest.error import SourceConnectionError
+from unstructured.ingest.interfaces import SourceMetadata
 from unstructured.ingest.logger import logger
 from unstructured.utils import requires_dependencies
 
@@ -52,23 +54,69 @@ class GitHubIngestDoc(GitIngestDoc):
     connector_config: SimpleGitHubConfig
     registry_name: str = "github"
 
-    def _fetch_and_write(self) -> None:
-        content_file = self.connector_config.get_repo().get_contents(self.path)
+    @property
+    def date_created(self) -> t.Optional[str]:
+        return None
+
+    @requires_dependencies(["github"], extras="github")
+    def _fetch_file(self):
+        from github.GithubException import UnknownObjectException
+
+        try:
+            content_file = self.connector_config.get_repo().get_contents(self.path)
+        except UnknownObjectException:
+            logger.error(f"File doesn't exists {self.connector_config.url}/{self.path}")
+            return None
+
+        return content_file
+
+    def _fetch_content(self, content_file):
         contents = b""
         if (
             not content_file.content  # type: ignore
             and content_file.encoding == "none"  # type: ignore
             and content_file.size  # type: ignore
         ):
             logger.info("File too large for the GitHub API, using direct download link instead.")
+            # NOTE: Maybe add a raise_for_status to catch connection timeout or HTTP Errors?
             response = requests.get(content_file.download_url)  # type: ignore
             if response.status_code != 200:
                 logger.info("Direct download link has failed... Skipping this file.")
+                return None
             else:
                 contents = response.content
         else:
             contents = content_file.decoded_content  # type: ignore
+        return contents
 
+    def update_source_metadata(self, **kwargs):
+        content_file = kwargs.get("content_file", self._fetch_file())
+        if content_file is None:
+            self.source_metadata = SourceMetadata(
+                exists=False,
+            )
+            return
+
+        date_modified = datetime.strptime(
+            content_file.last_modified,
+            "%a, %d %b %Y %H:%M:%S %Z",
+        ).isoformat()
+        self.source_metadata = SourceMetadata(
+            date_modified=date_modified,
+            version=content_file.etag,
+            source_url=content_file.download_url,
+            exists=True,
+        )
+
+    def _fetch_and_write(self) -> None:
+        content_file = self._fetch_file()
+        self.update_source_metadata(content_file=content_file)
+        contents = self._fetch_content(content_file)
+        if contents is None:
+            raise ValueError(
+                f"Failed to retrieve file from repo "
+                f"{self.connector_config.url}/{self.path}. Check logs",
+            )
         with open(self.filename, "wb") as f:
             f.write(contents)